Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 08:56:37.763072 21675 test_util.cc:274] Using random seed: -2123341970
2025-01-12T08:56:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T08:56:37Z Disabled control of system clock
I20250112 08:56:37.831146 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:45457
--webserver_interface=127.21.42.254
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 08:56:38.130004 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:38.130582 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:38.131031 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:56:38.161998 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:56:38.162331 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:38.162578 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:56:38.162819 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:56:38.196573 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.42.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:38.197952 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:38.199496 21684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:38.214819 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:56:38.215083 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:38.216938 21684 server_base.cc:1034] running on GCE node
W20250112 08:56:38.216223 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:39.383944 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:39.386618 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:39.388000 21684 hybrid_clock.cc:648] HybridClock initialized: now 1736672199387959 us; error 57 us; skew 500 ppm
I20250112 08:56:39.388973 21684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:39.395462 21684 webserver.cc:458] Webserver started at http://127.21.42.254:40287/ using document root <none> and password file <none>
I20250112 08:56:39.396432 21684 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:39.396648 21684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:39.397096 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:39.402062 21684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data/instance:
uuid: "15d38686b2c64b20ba87275e3ba176ab"
format_stamp: "Formatted at 2025-01-12 08:56:39 on dist-test-slave-ghv6"
I20250112 08:56:39.403154 21684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal/instance:
uuid: "15d38686b2c64b20ba87275e3ba176ab"
format_stamp: "Formatted at 2025-01-12 08:56:39 on dist-test-slave-ghv6"
I20250112 08:56:39.410310 21684 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250112 08:56:39.416567 21700 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:39.417624 21684 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250112 08:56:39.417933 21684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal
uuid: "15d38686b2c64b20ba87275e3ba176ab"
format_stamp: "Formatted at 2025-01-12 08:56:39 on dist-test-slave-ghv6"
I20250112 08:56:39.418243 21684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:39.469753 21684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:39.471191 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:39.471629 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:39.541828 21684 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.254:45457
I20250112 08:56:39.541922 21751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.254:45457 every 8 connection(s)
I20250112 08:56:39.544533 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
I20250112 08:56:39.549737 21752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:39.554862 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 21684
I20250112 08:56:39.555271 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-0/wal/instance
I20250112 08:56:39.558837 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.253:41411
--webserver_interface=127.21.42.253
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:56:39.560715 21752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:39.585827 21752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:56:39.588079 21703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
W20250112 08:56:39.591984 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
I20250112 08:56:39.641506 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 1
W20250112 08:56:39.645300 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
I20250112 08:56:39.713804 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 2
W20250112 08:56:39.717567 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
I20250112 08:56:39.809062 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 3
W20250112 08:56:39.812893 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
W20250112 08:56:39.863504 21754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:39.864106 21754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:39.864574 21754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:56:39.894382 21754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:56:39.894649 21754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:39.894908 21754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:56:39.895121 21754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:56:39.930295 21754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.253:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.42.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:39.931603 21754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:39.933166 21754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:39.949658 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:39.956353 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 4
W20250112 08:56:39.961010 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
I20250112 08:56:40.260427 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 5
W20250112 08:56:40.265831 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
I20250112 08:56:40.813212 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 6
W20250112 08:56:40.821120 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:41411: Network error: Client connection negotiation failed: client connection to 127.21.42.253:41411: connect: Connection refused (error 111)
W20250112 08:56:39.950415 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:39.952042 21754 server_base.cc:1034] running on GCE node
W20250112 08:56:39.950899 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:41.138391 21754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:41.140988 21754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:41.142359 21754 hybrid_clock.cc:648] HybridClock initialized: now 1736672201142310 us; error 59 us; skew 500 ppm
I20250112 08:56:41.143191 21754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:41.149787 21754 webserver.cc:458] Webserver started at http://127.21.42.253:42263/ using document root <none> and password file <none>
I20250112 08:56:41.150720 21754 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:41.150936 21754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:41.151371 21754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:41.156626 21754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data/instance:
uuid: "5fc0c669abed404c80d7c4cf4769f55a"
format_stamp: "Formatted at 2025-01-12 08:56:41 on dist-test-slave-ghv6"
I20250112 08:56:41.157814 21754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal/instance:
uuid: "5fc0c669abed404c80d7c4cf4769f55a"
format_stamp: "Formatted at 2025-01-12 08:56:41 on dist-test-slave-ghv6"
I20250112 08:56:41.165110 21754 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250112 08:56:41.170610 21775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:41.171535 21754 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250112 08:56:41.171866 21754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal
uuid: "5fc0c669abed404c80d7c4cf4769f55a"
format_stamp: "Formatted at 2025-01-12 08:56:41 on dist-test-slave-ghv6"
I20250112 08:56:41.172219 21754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:41.220242 21754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:41.221635 21754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:41.222076 21754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:41.290331 21754 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.253:41411
I20250112 08:56:41.290414 21826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.253:41411 every 8 connection(s)
I20250112 08:56:41.293076 21754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
I20250112 08:56:41.298071 21827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:41.301695 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 21754
I20250112 08:56:41.302062 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-1/wal/instance
I20250112 08:56:41.304283 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:43161
--webserver_interface=127.21.42.252
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:56:41.309044 21827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:41.335896 21827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:41.349581 21827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:56:41.352243 21779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
W20250112 08:56:41.356452 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:41.405882 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 1
W20250112 08:56:41.409631 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:41.477993 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 2
W20250112 08:56:41.481755 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:41.573139 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 3
W20250112 08:56:41.576817 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
W20250112 08:56:41.618492 21829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:41.619082 21829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:41.619516 21829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:56:41.650231 21829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:56:41.650627 21829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:41.650893 21829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:56:41.651130 21829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:56:41.686062 21829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.42.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:41.687371 21829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:41.688982 21829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:41.703377 21840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:41.720160 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 4
W20250112 08:56:41.724465 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:41.881706 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } attempt: 7
I20250112 08:56:41.893281 21752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:56:41.898716 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:41.957337 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 1
W20250112 08:56:41.961648 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.023826 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 5
W20250112 08:56:42.027968 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.043251 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 2
W20250112 08:56:42.047696 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.133335 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 3
W20250112 08:56:42.137818 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.278508 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 4
W20250112 08:56:42.283193 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.566848 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 5
W20250112 08:56:42.572319 21752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:42.575325 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 6
W20250112 08:56:42.582460 21827 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:43161: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
W20250112 08:56:41.705194 21839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:41.705600 21829 server_base.cc:1034] running on GCE node
W20250112 08:56:41.704069 21842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:42.911154 21829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:42.913700 21829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:42.915063 21829 hybrid_clock.cc:648] HybridClock initialized: now 1736672202915025 us; error 52 us; skew 500 ppm
I20250112 08:56:42.915868 21829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:42.927273 21829 webserver.cc:458] Webserver started at http://127.21.42.252:36567/ using document root <none> and password file <none>
I20250112 08:56:42.928256 21829 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:42.928490 21829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:42.928922 21829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:42.933741 21829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/instance:
uuid: "caa33dde8b5248e7ab4d6087d8e9d801"
format_stamp: "Formatted at 2025-01-12 08:56:42 on dist-test-slave-ghv6"
I20250112 08:56:42.934849 21829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal/instance:
uuid: "caa33dde8b5248e7ab4d6087d8e9d801"
format_stamp: "Formatted at 2025-01-12 08:56:42 on dist-test-slave-ghv6"
I20250112 08:56:42.942176 21829 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250112 08:56:42.948346 21854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:42.949455 21829 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250112 08:56:42.949791 21829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
uuid: "caa33dde8b5248e7ab4d6087d8e9d801"
format_stamp: "Formatted at 2025-01-12 08:56:42 on dist-test-slave-ghv6"
I20250112 08:56:42.950134 21829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:43.004750 21829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:43.006213 21829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:43.006651 21829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:43.082309 21829 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.252:43161
I20250112 08:56:43.082919 21905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.252:43161 every 8 connection(s)
I20250112 08:56:43.085108 21829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
I20250112 08:56:43.090977 21906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:43.091727 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 21829
I20250112 08:56:43.092238 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal/instance
I20250112 08:56:43.110437 21906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:43.124922 21752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 6
I20250112 08:56:43.143334 21906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:43.157889 21906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:56:43.164222 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Bootstrap starting.
I20250112 08:56:43.171849 21752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:43.174260 21752 log.cc:826] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Log is configured to *not* fsync() on all Append() calls
I20250112 08:56:43.177290 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Bootstrap starting.
I20250112 08:56:43.181607 21752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: No bootstrap required, opened a new log
I20250112 08:56:43.185916 21906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:43.188902 21906 log.cc:826] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Log is configured to *not* fsync() on all Append() calls
I20250112 08:56:43.194303 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: No bootstrap required, opened a new log
I20250112 08:56:43.213764 21752 raft_consensus.cc:357] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.214433 21752 raft_consensus.cc:383] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:43.214661 21752 raft_consensus.cc:738] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d38686b2c64b20ba87275e3ba176ab, State: Initialized, Role: FOLLOWER
I20250112 08:56:43.215368 21752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.218705 21919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.219595 21919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.220480 21752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:56:43.222663 21906 raft_consensus.cc:357] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.223383 21906 raft_consensus.cc:383] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:43.223619 21906 raft_consensus.cc:738] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa33dde8b5248e7ab4d6087d8e9d801, State: Initialized, Role: FOLLOWER
I20250112 08:56:43.224416 21906 consensus_queue.cc:260] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.227533 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.228487 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.229573 21906 sys_catalog.cc:564] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 08:56:43.230683 21706 tablet.cc:2367] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 08:56:43.262023 21860 tablet.cc:2367] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 08:56:43.266813 21938 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:56:43.267172 21938 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:56:43.290781 21919 raft_consensus.cc:491] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:43.291343 21919 raft_consensus.cc:513] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.296753 21881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15d38686b2c64b20ba87275e3ba176ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:43.298250 21881 raft_consensus.cc:2463] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d38686b2c64b20ba87275e3ba176ab in term 0.
I20250112 08:56:43.268213 21684 master_runner.cc:186] Error getting master registration for 127.21.42.253:41411: OK, instance_id { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" instance_seqno: 1736672201271662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 08:56:43.304566 21704 leader_election.cc:304] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801; no voters:
I20250112 08:56:43.304992 21684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:56:43.307057 21919 leader_election.cc:290] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:43.308316 21919 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:43.309128 21919 raft_consensus.cc:491] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:43.309932 21919 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:43.324122 21919 raft_consensus.cc:513] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.326330 21919 leader_election.cc:290] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [CANDIDATE]: Term 1 election: Requested vote from peers 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:43.333606 21881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15d38686b2c64b20ba87275e3ba176ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801"
I20250112 08:56:43.334203 21881 raft_consensus.cc:3054] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:56:43.336012 21943 catalog_manager.cc:1559] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:56:43.336413 21943 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:56:43.341732 21881 raft_consensus.cc:2463] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d38686b2c64b20ba87275e3ba176ab in term 1.
I20250112 08:56:43.342871 21704 leader_election.cc:304] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801; no voters:
I20250112 08:56:43.343772 21919 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:43.344321 21919 raft_consensus.cc:695] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [term 1 LEADER]: Becoming Leader. State: Replica: 15d38686b2c64b20ba87275e3ba176ab, State: Running, Role: LEADER
I20250112 08:56:43.345808 21919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.354914 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d38686b2c64b20ba87275e3ba176ab. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.355790 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: LEADER
I20250112 08:56:43.366173 21947 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:56:43.374861 21947 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:56:43.278338 21829 master_runner.cc:186] Error getting master registration for 127.21.42.253:41411: OK, instance_id { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" instance_seqno: 1736672201271662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 08:56:43.383406 21829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:56:43.405593 21881 raft_consensus.cc:1270] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Refusing update from remote peer 15d38686b2c64b20ba87275e3ba176ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:43.407699 21919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:43.445107 21948 mvcc.cc:204] Tried to move back new op lower bound from 7113409345126612992 to 7113409344916455424. Current Snapshot: MvccSnapshot[applied={T|T < 7113409345126612992}]
I20250112 08:56:43.447311 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d38686b2c64b20ba87275e3ba176ab. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.451083 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.453372 21945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.454378 21945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: LEADER
I20250112 08:56:43.470057 21947 catalog_manager.cc:1348] Generated new cluster ID: 99ee22f2d13d4f5080e046f96817436c
I20250112 08:56:43.470396 21947 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:56:43.475456 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.476294 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.512357 21947 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 08:56:43.514559 21947 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:56:43.555051 21947 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Generated new TSK 0
I20250112 08:56:43.556293 21947 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:56:43.579178 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.193:0
--local_ip_for_outbound_sockets=127.21.42.193
--webserver_interface=127.21.42.193
--webserver_port=0
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:56:43.643062 21827 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } attempt: 7
I20250112 08:56:43.683429 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: Bootstrap starting.
I20250112 08:56:43.700146 21827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:43.705932 21827 log.cc:826] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: Log is configured to *not* fsync() on all Append() calls
I20250112 08:56:43.724426 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: No bootstrap required, opened a new log
I20250112 08:56:43.790797 21827 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.792789 21827 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:43.793509 21827 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc0c669abed404c80d7c4cf4769f55a, State: Initialized, Role: FOLLOWER
I20250112 08:56:43.795598 21827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:43.814684 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15d38686b2c64b20ba87275e3ba176ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a"
I20250112 08:56:43.816025 21801 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:43.820995 21802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15d38686b2c64b20ba87275e3ba176ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
W20250112 08:56:43.822062 21802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 2) took 527 ms (client timeout 641 ms). Trace:
W20250112 08:56:43.822396 21802 rpcz_store.cc:269] 0112 08:56:43.294604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:43.294750 (+ 146us) service_pool.cc:225] Handling call
0112 08:56:43.821962 (+527212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 08:56:43.826331 21801 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d38686b2c64b20ba87275e3ba176ab in term 1.
I20250112 08:56:43.827569 21800 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Refusing update from remote peer 15d38686b2c64b20ba87275e3ba176ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:43.835321 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.836050 21919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:43.836452 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.840005 21827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:56:43.910303 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d38686b2c64b20ba87275e3ba176ab. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.910550 21962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:43.911072 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:43.911373 21962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:56:44.005203 21973 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250112 08:56:44.110127 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:44.110883 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:44.111972 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:56:44.169979 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:44.171672 21954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.193
I20250112 08:56:44.233652 21954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.42.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.193
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:44.235626 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:44.237732 21954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:44.258864 21979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:44.341688 21962 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:44.342360 21962 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:44.346788 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:44.351789 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:44.354051 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:44.355307 21957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:44.355727 21962 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:44.360071 21943 catalog_manager.cc:1260] Loaded cluster ID: 99ee22f2d13d4f5080e046f96817436c
I20250112 08:56:44.361140 21943 catalog_manager.cc:1553] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: loading cluster ID for follower catalog manager: success
I20250112 08:56:44.371280 21943 catalog_manager.cc:1575] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: acquiring CA information for follower catalog manager: success
I20250112 08:56:44.375612 21919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:44.376379 21919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: LEADER
I20250112 08:56:44.393002 21943 catalog_manager.cc:1603] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:56:44.576709 21987 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:44.577608 21987 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:44.580942 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:44.582742 21987 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:44.583117 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:44.585666 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:44.587164 21987 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:45.029664 21973 catalog_manager.cc:1260] Loaded cluster ID: 99ee22f2d13d4f5080e046f96817436c
I20250112 08:56:45.030053 21973 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: loading cluster ID for follower catalog manager: success
I20250112 08:56:45.061785 21973 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: acquiring CA information for follower catalog manager: success
I20250112 08:56:45.069888 21973 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:56:45.807608 21989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:45.808374 21989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:45.812033 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:45.815714 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:45.818241 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:45.823659 21989 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:45.824851 21989 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:46.035106 21991 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:46.035970 21991 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:46.049162 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:46.053862 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:46.056915 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:46.059248 21991 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:46.060350 21991 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:46.405859 21993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
W20250112 08:56:45.655673 21978 debug-util.cc:398] Leaking SignalData structure 0x7b080001c400 after lost signal to thread 21954
W20250112 08:56:46.409080 21978 kernel_stack_watchdog.cc:198] Thread 21954 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250112 08:56:46.410236 21954 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 2.150s user 0.357s sys 0.733s
W20250112 08:56:44.261441 21980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:46.410862 21993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
W20250112 08:56:46.411675 21982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:56:46.411655 21954 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 2.150s user 0.357s sys 0.733s
I20250112 08:56:46.414355 21993 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:46.414603 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:46.416435 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
W20250112 08:56:46.417586 21981 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2156 milliseconds
I20250112 08:56:46.417917 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:46.418149 21954 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 08:56:46.418994 21993 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:46.419521 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:46.422873 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:46.424357 21954 hybrid_clock.cc:648] HybridClock initialized: now 1736672206424310 us; error 38 us; skew 500 ppm
I20250112 08:56:46.425494 21954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:46.456652 21954 webserver.cc:458] Webserver started at http://127.21.42.193:41789/ using document root <none> and password file <none>
I20250112 08:56:46.458063 21954 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:46.458424 21954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:46.459048 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:46.466274 21954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data/instance:
uuid: "76718d1596134c83900541c6d398d59e"
format_stamp: "Formatted at 2025-01-12 08:56:46 on dist-test-slave-ghv6"
I20250112 08:56:46.467870 21954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/instance:
uuid: "76718d1596134c83900541c6d398d59e"
format_stamp: "Formatted at 2025-01-12 08:56:46 on dist-test-slave-ghv6"
I20250112 08:56:46.487450 21954 fs_manager.cc:696] Time spent creating directory manager: real 0.019s user 0.006s sys 0.011s
I20250112 08:56:46.504242 21998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:46.505776 21954 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.010s sys 0.002s
I20250112 08:56:46.506290 21954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal
uuid: "76718d1596134c83900541c6d398d59e"
format_stamp: "Formatted at 2025-01-12 08:56:46 on dist-test-slave-ghv6"
I20250112 08:56:46.507037 21954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:46.637732 21954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:46.639707 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:46.640385 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:46.651301 21954 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:56:46.671319 22004 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:46.673154 21954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:56:46.673697 22004 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:46.674464 21954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250112 08:56:46.674813 21954 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:56:46.675050 21954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:46.677361 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:46.679600 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:46.681010 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:46.686764 22004 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:46.687594 22004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:46.958640 21954 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.193:37233
I20250112 08:56:46.958773 22112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.193:37233 every 8 connection(s)
I20250112 08:56:46.962163 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
I20250112 08:56:46.972698 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 21954
I20250112 08:56:46.973246 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/instance
I20250112 08:56:46.988463 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.194:0
--local_ip_for_outbound_sockets=127.21.42.194
--webserver_interface=127.21.42.194
--webserver_port=0
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:56:47.049340 22113 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
I20250112 08:56:47.049845 22113 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:47.051177 22113 heartbeater.cc:510] Master 127.21.42.252:43161 requested a full tablet report, sending...
I20250112 08:56:47.053562 22114 heartbeater.cc:346] Connected to a master server at 127.21.42.253:41411
I20250112 08:56:47.053859 22114 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:47.054697 22114 heartbeater.cc:510] Master 127.21.42.253:41411 requested a full tablet report, sending...
I20250112 08:56:47.055047 21871 ts_manager.cc:194] Registered new tserver with Master: 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:47.058019 21792 ts_manager.cc:194] Registered new tserver with Master: 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:47.060155 22117 heartbeater.cc:346] Connected to a master server at 127.21.42.254:45457
I20250112 08:56:47.060444 22117 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:47.061280 22117 heartbeater.cc:510] Master 127.21.42.254:45457 requested a full tablet report, sending...
I20250112 08:56:47.063877 21716 ts_manager.cc:194] Registered new tserver with Master: 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:47.066519 21716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.193:36265
W20250112 08:56:47.401247 22120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:47.401834 22120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:47.402458 22120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 08:56:47.426491 22128 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:47.426985 22128 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:47.430342 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:47.430430 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:47.431872 22128 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:47.432188 21778 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:47.432937 22128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:56:47.440160 22120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:47.441094 22120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.194
I20250112 08:56:47.476791 22120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.42.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.194
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:47.478140 22120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:47.479684 22120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:47.499644 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:47.570139 22117 heartbeater.cc:502] Master 127.21.42.254:45457 was elected leader, sending a full tablet report...
I20250112 08:56:48.044098 22137 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:48.047313 22137 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:48.053159 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:48.056866 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:48.058326 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:48.060807 22137 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:48.061596 22137 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:48.604672 22140 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:48.605396 22140 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:48.609998 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:48.610508 22140 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:48.612222 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:48.614238 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:48.615559 22140 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:56:47.505944 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:56:47.500448 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:56:48.885326 22132 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1383 milliseconds
I20250112 08:56:48.885447 22120 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 08:56:48.886665 22120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:48.888849 22120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:48.890211 22120 hybrid_clock.cc:648] HybridClock initialized: now 1736672208890181 us; error 47 us; skew 500 ppm
I20250112 08:56:48.890993 22120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:48.902665 22120 webserver.cc:458] Webserver started at http://127.21.42.194:37145/ using document root <none> and password file <none>
I20250112 08:56:48.903614 22120 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:48.903824 22120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:48.904347 22120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:48.909281 22120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data/instance:
uuid: "53068c99960647728141706e9c50b870"
format_stamp: "Formatted at 2025-01-12 08:56:48 on dist-test-slave-ghv6"
I20250112 08:56:48.910418 22120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/instance:
uuid: "53068c99960647728141706e9c50b870"
format_stamp: "Formatted at 2025-01-12 08:56:48 on dist-test-slave-ghv6"
I20250112 08:56:48.917376 22120 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250112 08:56:48.922905 22145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:48.924084 22120 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250112 08:56:48.924410 22120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal
uuid: "53068c99960647728141706e9c50b870"
format_stamp: "Formatted at 2025-01-12 08:56:48 on dist-test-slave-ghv6"
I20250112 08:56:48.924746 22120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:48.977598 22120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:48.979060 22120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:48.979516 22120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:48.982096 22120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:56:48.986802 22120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:56:48.987010 22120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:48.987249 22120 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:56:48.987409 22120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:49.092475 22220 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:49.093124 22220 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:49.096431 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:49.096720 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:49.098587 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:49.106622 22220 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:49.107735 22220 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:49.128649 22236 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:49.128999 22236 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:49.130775 22236 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:49.132295 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:49.132413 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:49.133762 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:49.134604 22236 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:49.164108 22120 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.194:34135
I20250112 08:56:49.164211 22260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.194:34135 every 8 connection(s)
I20250112 08:56:49.166877 22120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
I20250112 08:56:49.177337 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 22120
I20250112 08:56:49.178086 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/instance
I20250112 08:56:49.201361 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.195:0
--local_ip_for_outbound_sockets=127.21.42.195
--webserver_interface=127.21.42.195
--webserver_port=0
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:56:49.212774 22262 heartbeater.cc:346] Connected to a master server at 127.21.42.253:41411
I20250112 08:56:49.213279 22262 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:49.214668 22262 heartbeater.cc:510] Master 127.21.42.253:41411 requested a full tablet report, sending...
I20250112 08:56:49.214843 22263 heartbeater.cc:346] Connected to a master server at 127.21.42.254:45457
I20250112 08:56:49.215214 22263 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:49.216184 22261 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
I20250112 08:56:49.216200 22263 heartbeater.cc:510] Master 127.21.42.254:45457 requested a full tablet report, sending...
I20250112 08:56:49.216606 22261 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:49.217473 22261 heartbeater.cc:510] Master 127.21.42.252:43161 requested a full tablet report, sending...
I20250112 08:56:49.218262 21792 ts_manager.cc:194] Registered new tserver with Master: 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:49.220134 21871 ts_manager.cc:194] Registered new tserver with Master: 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:49.220067 21716 ts_manager.cc:194] Registered new tserver with Master: 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:49.222271 21716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.194:55563
W20250112 08:56:49.521404 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:49.521931 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:49.522496 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:56:49.553655 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:49.554605 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.195
I20250112 08:56:49.589362 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.42.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.195
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:49.590720 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:49.592270 22271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:56:49.611785 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:49.724992 22263 heartbeater.cc:502] Master 127.21.42.254:45457 was elected leader, sending a full tablet report...
I20250112 08:56:50.107890 22284 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:50.108299 22284 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:50.109730 22284 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:56:50.110997 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:50.111195 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:56:50.113087 21857 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:56:50.113626 22284 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:56:49.611979 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:56:49.612771 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:49.614295 22271 server_base.cc:1034] running on GCE node
I20250112 08:56:50.825119 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:56:50.827338 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:56:50.828678 22271 hybrid_clock.cc:648] HybridClock initialized: now 1736672210828654 us; error 41 us; skew 500 ppm
I20250112 08:56:50.829548 22271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:56:50.835744 22271 webserver.cc:458] Webserver started at http://127.21.42.195:34895/ using document root <none> and password file <none>
I20250112 08:56:50.836694 22271 fs_manager.cc:362] Metadata directory not provided
I20250112 08:56:50.836907 22271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:56:50.837350 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:56:50.842641 22271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data/instance:
uuid: "8c608785d6874abf9cc28db11a979ddd"
format_stamp: "Formatted at 2025-01-12 08:56:50 on dist-test-slave-ghv6"
I20250112 08:56:50.843658 22271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/instance:
uuid: "8c608785d6874abf9cc28db11a979ddd"
format_stamp: "Formatted at 2025-01-12 08:56:50 on dist-test-slave-ghv6"
I20250112 08:56:50.850710 22271 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250112 08:56:50.856915 22289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:50.857933 22271 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250112 08:56:50.858233 22271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal
uuid: "8c608785d6874abf9cc28db11a979ddd"
format_stamp: "Formatted at 2025-01-12 08:56:50 on dist-test-slave-ghv6"
I20250112 08:56:50.858538 22271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:56:50.911197 22271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:56:50.913378 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:56:50.914060 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:56:50.917595 22271 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:56:50.922080 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:56:50.922297 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:50.922560 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:56:50.922721 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:56:50.923864 22296 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:56:50.924536 22296 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:56:50.926855 22296 leader_election.cc:290] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161)
I20250112 08:56:50.927467 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:56:50.928385 21880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5fc0c669abed404c80d7c4cf4769f55a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" is_pre_election: true
I20250112 08:56:50.929888 21779 leader_election.cc:304] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5fc0c669abed404c80d7c4cf4769f55a; no voters: 15d38686b2c64b20ba87275e3ba176ab, caa33dde8b5248e7ab4d6087d8e9d801
I20250112 08:56:50.930558 22296 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5fc0c669abed404c80d7c4cf4769f55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:56:51.083201 22271 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.195:33805
I20250112 08:56:51.083289 22402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.195:33805 every 8 connection(s)
I20250112 08:56:51.086685 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
I20250112 08:56:51.097398 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 22271
I20250112 08:56:51.097851 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/instance
I20250112 08:56:51.142190 22403 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
I20250112 08:56:51.142697 22403 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:51.144331 22403 heartbeater.cc:510] Master 127.21.42.252:43161 requested a full tablet report, sending...
I20250112 08:56:51.147825 21871 ts_manager.cc:194] Registered new tserver with Master: 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:51.156229 22405 heartbeater.cc:346] Connected to a master server at 127.21.42.254:45457
I20250112 08:56:51.156574 22405 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:51.156939 22404 heartbeater.cc:346] Connected to a master server at 127.21.42.253:41411
I20250112 08:56:51.157222 22404 heartbeater.cc:463] Registering TS with master...
I20250112 08:56:51.157511 22405 heartbeater.cc:510] Master 127.21.42.254:45457 requested a full tablet report, sending...
I20250112 08:56:51.158035 22404 heartbeater.cc:510] Master 127.21.42.253:41411 requested a full tablet report, sending...
I20250112 08:56:51.160022 21716 ts_manager.cc:194] Registered new tserver with Master: 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:51.160357 21792 ts_manager.cc:194] Registered new tserver with Master: 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:51.161677 21716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.195:37593
I20250112 08:56:51.167593 21675 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 08:56:51.221700 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_c921c9a4fd0c4917adf468321d5d99ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:51.222375 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_09fe45635da94ef28b689201b68b14e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:51.224985 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_583b082367b54d91a181f63ec3131b08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.226951 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583b082367b54d91a181f63ec3131b08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:51.227432 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c921c9a4fd0c4917adf468321d5d99ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:51.227280 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09fe45635da94ef28b689201b68b14e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.230768 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_d1b3a19f0fd74a94b8f0bba06822ea98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.232820 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b3a19f0fd74a94b8f0bba06822ea98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.308524 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_c921c9a4fd0c4917adf468321d5d99ee" } new_table_name: "default.Table_144c1527f47e4533848766fc39213caa" modify_external_catalogs: true
I20250112 08:56:51.354488 21717 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_583b082367b54d91a181f63ec3131b08" } modify_external_catalogs: true
I20250112 08:56:51.355301 21717 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_583b082367b54d91a181f63ec3131b08" } modify_external_catalogs: true
I20250112 08:56:51.356884 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_09fe45635da94ef28b689201b68b14e3" } new_table_name: "default.Table_953942deee4841b9ae8aa1e6d76485d2" modify_external_catalogs: true
I20250112 08:56:51.359695 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d1b3a19f0fd74a94b8f0bba06822ea98" } modify_external_catalogs: true
I20250112 08:56:51.363061 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d1b3a19f0fd74a94b8f0bba06822ea98" } modify_external_catalogs: true
W20250112 08:56:51.380628 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c73ff6e4fdf46199f6aa3a04b72a34a has no consensus state
W20250112 08:56:51.387459 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48b63eff49fc4b829afc8160b266fee5 consensus state has no leader
I20250112 08:56:51.389503 21714 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4946d19ffbdf4c43905658588a357c3b
I20250112 08:56:51.400915 21717 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4223a3de54664e55a10ec3a82fa09ebc
I20250112 08:56:51.409097 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 21829
W20250112 08:56:51.435999 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_953942deee4841b9ae8aa1e6d76485d2 [id=dbe51ac33754416f84c887ba5a2af3d6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2c73ff6e4fdf46199f6aa3a04b72a34a has no consensus state
I20250112 08:56:51.448807 22047 tablet_service.cc:1467] Processing CreateTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (DEFAULT_TABLE table=default.Table_953942deee4841b9ae8aa1e6d76485d2 [id=dbe51ac33754416f84c887ba5a2af3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.450456 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.447434 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 48b63eff49fc4b829afc8160b266fee5 (DEFAULT_TABLE table=default.Table_c921c9a4fd0c4917adf468321d5d99ee [id=384bd09cd5264829a09783af5539802a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.461242 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (DEFAULT_TABLE table=default.Table_953942deee4841b9ae8aa1e6d76485d2 [id=dbe51ac33754416f84c887ba5a2af3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.463280 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.464761 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b63eff49fc4b829afc8160b266fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.465945 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 48b63eff49fc4b829afc8160b266fee5 (DEFAULT_TABLE table=default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.467131 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b63eff49fc4b829afc8160b266fee5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:51.469838 21704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.42.252:43161 (error 108) [suppressed 12 similar messages]
W20250112 08:56:51.501468 21704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab -> Peer caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161): Couldn't send request to peer caa33dde8b5248e7ab4d6087d8e9d801. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:51.507315 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (DEFAULT_TABLE table=default.Table_953942deee4841b9ae8aa1e6d76485d2 [id=dbe51ac33754416f84c887ba5a2af3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.509341 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.509450 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_9ea82d5fdf6f4783964e0f0d6135253f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.511072 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ea82d5fdf6f4783964e0f0d6135253f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.512369 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 48b63eff49fc4b829afc8160b266fee5 (DEFAULT_TABLE table=default.Table_c921c9a4fd0c4917adf468321d5d99ee [id=384bd09cd5264829a09783af5539802a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.514571 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b63eff49fc4b829afc8160b266fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.529527 22456 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:51.530833 22458 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:51.537920 22458 tablet_bootstrap.cc:654] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.538578 22456 tablet_bootstrap.cc:654] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.541922 22456 log.cc:826] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Log is configured to *not* fsync() on all Append() calls
I20250112 08:56:51.542479 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_12e368eed3384bfe83c62be5a7aac80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:51.543718 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_8e7d87182b7f47f088f746440d3bcc19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:51.549556 22458 log.cc:826] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Log is configured to *not* fsync() on all Append() calls
W20250112 08:56:51.551553 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e7d87182b7f47f088f746440d3bcc19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:51.552103 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12e368eed3384bfe83c62be5a7aac80a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.562736 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_771927535b7f4972ae195b3702a277b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.564502 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_771927535b7f4972ae195b3702a277b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.583781 22458 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:51.584501 22458 ts_tablet_manager.cc:1397] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.020s
I20250112 08:56:51.596490 22461 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:56:51.616915 22113 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:43161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:51.618810 22461 tablet_bootstrap.cc:654] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.621273 22461 log.cc:826] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Log is configured to *not* fsync() on all Append() calls
I20250112 08:56:51.627936 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_9ea82d5fdf6f4783964e0f0d6135253f" } new_table_name: "default.Table_2c562866672a4bbfa557feddf8f6464e" modify_external_catalogs: true
I20250112 08:56:51.637380 22458 raft_consensus.cc:357] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:51.638310 22458 raft_consensus.cc:383] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.638598 22458 raft_consensus.cc:738] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.639523 22458 consensus_queue.cc:260] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:51.640295 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 8c934e1a62444e64911a279f6a544c96 (DEFAULT_TABLE table=default.Table_9ea82d5fdf6f4783964e0f0d6135253f [id=8196e2c3275a4c3d9ce4a6ed3b997448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.641422 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c934e1a62444e64911a279f6a544c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.645640 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 8c934e1a62444e64911a279f6a544c96 (DEFAULT_TABLE table=default.Table_9ea82d5fdf6f4783964e0f0d6135253f [id=8196e2c3275a4c3d9ce4a6ed3b997448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.646729 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c934e1a62444e64911a279f6a544c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.647538 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 8c934e1a62444e64911a279f6a544c96 (DEFAULT_TABLE table=default.Table_9ea82d5fdf6f4783964e0f0d6135253f [id=8196e2c3275a4c3d9ce4a6ed3b997448]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.648577 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c934e1a62444e64911a279f6a544c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.648361 21717 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_144c1527f47e4533848766fc39213caa" } modify_external_catalogs: true
I20250112 08:56:51.648924 21717 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_144c1527f47e4533848766fc39213caa" } modify_external_catalogs: true
I20250112 08:56:51.655308 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_953942deee4841b9ae8aa1e6d76485d2" } new_table_name: "default.Table_14794d2519464ce3bb6d9c0dc8f7758b" modify_external_catalogs: true
I20250112 08:56:51.657066 21715 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8e7d87182b7f47f088f746440d3bcc19" } modify_external_catalogs: true
I20250112 08:56:51.657665 21715 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8e7d87182b7f47f088f746440d3bcc19" } modify_external_catalogs: true
W20250112 08:56:51.659461 22403 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:43161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:51.664944 22405 heartbeater.cc:502] Master 127.21.42.254:45457 was elected leader, sending a full tablet report...
I20250112 08:56:51.675047 22458 ts_tablet_manager.cc:1428] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.090s user 0.037s sys 0.011s
I20250112 08:56:51.676695 22456 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:51.677260 22456 ts_tablet_manager.cc:1397] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.148s user 0.024s sys 0.012s
I20250112 08:56:51.678874 22458 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:51.685163 22458 tablet_bootstrap.cc:654] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:51.698098 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c934e1a62444e64911a279f6a544c96 consensus state has no leader
I20250112 08:56:51.702639 22461 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:51.703260 22461 ts_tablet_manager.cc:1397] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.007s
I20250112 08:56:51.718592 22458 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:51.719034 22458 ts_tablet_manager.cc:1397] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.026s
I20250112 08:56:51.721758 22458 raft_consensus.cc:357] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.722543 22458 raft_consensus.cc:383] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.722885 22458 raft_consensus.cc:738] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.723582 22458 consensus_queue.cc:260] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.735237 22458 ts_tablet_manager.cc:1428] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
I20250112 08:56:51.735378 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_d36a36d87f6a456b8922c5bc24659c21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.737571 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d36a36d87f6a456b8922c5bc24659c21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.743403 22458 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:51.749761 22458 tablet_bootstrap.cc:654] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.759872 22461 raft_consensus.cc:357] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.760928 22461 raft_consensus.cc:383] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.761272 22461 raft_consensus.cc:738] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.762004 22458 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:51.762562 22458 ts_tablet_manager.cc:1397] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250112 08:56:51.762471 22461 consensus_queue.cc:260] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.765846 22458 raft_consensus.cc:357] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.766683 22458 raft_consensus.cc:383] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.766992 22458 raft_consensus.cc:738] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.767864 22458 consensus_queue.cc:260] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.770098 22458 ts_tablet_manager.cc:1428] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250112 08:56:51.775319 22461 ts_tablet_manager.cc:1428] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.072s user 0.028s sys 0.016s
I20250112 08:56:51.779585 22461 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:51.779812 22456 raft_consensus.cc:357] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:51.780781 22456 raft_consensus.cc:383] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.781095 22456 raft_consensus.cc:738] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.782006 22456 consensus_queue.cc:260] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:51.788573 22121 tablet.cc:2367] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:56:51.792539 22461 tablet_bootstrap.cc:654] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.793684 21714 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_953942deee4841b9ae8aa1e6d76485d2 [id=dbe51ac33754416f84c887ba5a2af3d6]). Expected version 1 got 0
I20250112 08:56:51.794317 21714 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 48b63eff49fc4b829afc8160b266fee5 (table default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]). Expected version 1 got 0
I20250112 08:56:51.795058 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 (DEFAULT_TABLE table=default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.796209 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84f97537c74fb1ba32f7861bf27457. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:51.798529 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c73ff6e4fdf46199f6aa3a04b72a34a consensus state has no leader
I20250112 08:56:51.800118 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 48b63eff49fc4b829afc8160b266fee5 (table default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]). Expected version 1 got 0
I20250112 08:56:51.801664 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 (DEFAULT_TABLE table=default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.803246 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84f97537c74fb1ba32f7861bf27457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.805711 22456 ts_tablet_manager.cc:1428] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.128s user 0.032s sys 0.007s
I20250112 08:56:51.806746 22456 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:51.810048 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 (DEFAULT_TABLE table=default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.810714 22337 tablet_service.cc:1467] Processing CreateTablet for tablet e23582f78e674fd8b274669bc1e3a58a (DEFAULT_TABLE table=default.Table_12e368eed3384bfe83c62be5a7aac80a [id=1bbafad80c2f4de18948b31d817955c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.811146 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e84f97537c74fb1ba32f7861bf27457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.811686 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23582f78e674fd8b274669bc1e3a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.816675 22047 tablet_service.cc:1467] Processing CreateTablet for tablet e23582f78e674fd8b274669bc1e3a58a (DEFAULT_TABLE table=default.Table_12e368eed3384bfe83c62be5a7aac80a [id=1bbafad80c2f4de18948b31d817955c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:51.817454 22261 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:43161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111)
I20250112 08:56:51.817695 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23582f78e674fd8b274669bc1e3a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.817775 22192 tablet_service.cc:1467] Processing CreateTablet for tablet e23582f78e674fd8b274669bc1e3a58a (DEFAULT_TABLE table=default.Table_12e368eed3384bfe83c62be5a7aac80a [id=1bbafad80c2f4de18948b31d817955c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:51.818828 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e23582f78e674fd8b274669bc1e3a58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:51.826139 22456 tablet_bootstrap.cc:654] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.844271 22458 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:51.851130 22458 tablet_bootstrap.cc:654] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:51.853268 22407 tablet.cc:2367] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:56:51.857296 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b959aeabbb59496c93feb6f5fcc5b18c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:51.860558 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_7e088c3591104573aef358a41173f181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.862823 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e088c3591104573aef358a41173f181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:51.859696 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b959aeabbb59496c93feb6f5fcc5b18c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:51.869375 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 6e84f97537c74fb1ba32f7861bf27457
I20250112 08:56:51.873112 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:51.874146 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:51.875581 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:51.888257 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e03e9ba8aa6143669b27bd0f94949736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:51.891078 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e03e9ba8aa6143669b27bd0f94949736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:51.892570 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 6e84f97537c74fb1ba32f7861bf27457 because the tablet was not found. No further retry: Not found: Tablet not found: 6e84f97537c74fb1ba32f7861bf27457
I20250112 08:56:51.901129 22458 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:51.901561 22458 ts_tablet_manager.cc:1397] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.011s
I20250112 08:56:51.904706 22456 tablet_bootstrap.cc:492] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:51.905330 22456 ts_tablet_manager.cc:1397] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.011s
W20250112 08:56:51.909934 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 6e84f97537c74fb1ba32f7861bf27457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e84f97537c74fb1ba32f7861bf27457 already in progress: creating tablet
I20250112 08:56:51.909476 22456 raft_consensus.cc:357] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.910524 22456 raft_consensus.cc:383] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.910909 22456 raft_consensus.cc:738] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.911547 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d36a36d87f6a456b8922c5bc24659c21" } new_table_name: "default.Table_53ca25b8059f45ffa40dcfc1ed61793c" modify_external_catalogs: true
I20250112 08:56:51.911574 22456 consensus_queue.cc:260] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.912277 22439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:56:51.913138 22439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: LEADER
I20250112 08:56:51.913784 22456 ts_tablet_manager.cc:1428] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250112 08:56:51.914610 22456 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:51.919322 22458 raft_consensus.cc:357] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:51.920097 22458 raft_consensus.cc:383] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 08:56:51.920100 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 6e84f97537c74fb1ba32f7861bf27457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e84f97537c74fb1ba32f7861bf27457 already in progress: creating tablet
I20250112 08:56:51.920397 22458 raft_consensus.cc:738] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.921175 22458 consensus_queue.cc:260] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:51.923242 22456 tablet_bootstrap.cc:654] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.926699 22458 ts_tablet_manager.cc:1428] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20250112 08:56:51.930184 22458 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:51.936090 22458 tablet_bootstrap.cc:654] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:51.947328 22458 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:51.947775 22458 ts_tablet_manager.cc:1397] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.014s
W20250112 08:56:51.947948 22268 tablet.cc:2367] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:56:51.950524 22458 raft_consensus.cc:357] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.951323 22458 raft_consensus.cc:383] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:51.951632 22458 raft_consensus.cc:738] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:51.952622 22458 consensus_queue.cc:260] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.968227 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:51.973348 22458 ts_tablet_manager.cc:1428] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250112 08:56:51.974264 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b63eff49fc4b829afc8160b266fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:51.975158 22477 tablet_replica.cc:331] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:51.976281 22477 raft_consensus.cc:2238] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:51.976861 22477 raft_consensus.cc:2267] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:51.978302 22469 raft_consensus.cc:491] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:51.979089 22469 raft_consensus.cc:513] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:51.986145 22469 leader_election.cc:290] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:51.994092 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b63eff49fc4b829afc8160b266fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:51.995230 22475 tablet_replica.cc:331] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:51.996485 22475 raft_consensus.cc:2238] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:51.997155 22475 raft_consensus.cc:2267] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:51.987759 22477 ts_tablet_manager.cc:1905] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.006518 22475 ts_tablet_manager.cc:1905] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.020233 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b63eff49fc4b829afc8160b266fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:52.021780 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 48b63eff49fc4b829afc8160b266fee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48b63eff49fc4b829afc8160b266fee5 already in progress: creating tablet
I20250112 08:56:52.022413 21711 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_12e368eed3384bfe83c62be5a7aac80a" } new_table_name: "default.Table_b66f0362eec34c34a4efe32381f0d747" modify_external_catalogs: true
I20250112 08:56:52.028122 22461 tablet_bootstrap.cc:492] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:52.028631 22461 ts_tablet_manager.cc:1397] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.249s user 0.013s sys 0.004s
I20250112 08:56:52.031370 22461 raft_consensus.cc:357] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:52.034904 22461 raft_consensus.cc:383] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.035264 22461 raft_consensus.cc:738] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.037019 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0f3421a70fde4a968a22dcb2a2db7693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:52.038733 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f3421a70fde4a968a22dcb2a2db7693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.040128 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c73ff6e4fdf46199f6aa3a04b72a34a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:52.041029 22213 raft_consensus.cc:2463] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:52.042577 22290 leader_election.cc:304] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:52.045555 22469 raft_consensus.cc:2798] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:52.045918 22469 raft_consensus.cc:491] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:52.046247 22469 raft_consensus.cc:3054] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.049500 22461 consensus_queue.cc:260] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:52.053191 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c73ff6e4fdf46199f6aa3a04b72a34a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:52.054083 22068 raft_consensus.cc:2463] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:52.130522 22461 ts_tablet_manager.cc:1428] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.101s user 0.007s sys 0.000s
I20250112 08:56:52.130903 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_771927535b7f4972ae195b3702a277b4" } modify_external_catalogs: true
I20250112 08:56:52.131748 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_771927535b7f4972ae195b3702a277b4" } modify_external_catalogs: true
I20250112 08:56:52.139531 22461 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:52.154541 22456 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.155073 22456 ts_tablet_manager.cc:1397] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.241s user 0.071s sys 0.120s
I20250112 08:56:52.157799 22456 raft_consensus.cc:357] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.158550 22456 raft_consensus.cc:383] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.158829 22456 raft_consensus.cc:738] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.159505 22456 consensus_queue.cc:260] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.160470 22475 ts_tablet_manager.cc:1918] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.160876 22475 log.cc:1198] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:52.161916 22475 ts_tablet_manager.cc:1939] T 48b63eff49fc4b829afc8160b266fee5 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:52.162465 22456 ts_tablet_manager.cc:1428] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250112 08:56:52.167228 22456 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:52.175992 22456 tablet_bootstrap.cc:654] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.182754 22469 raft_consensus.cc:513] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.184949 22469 leader_election.cc:290] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:52.186966 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c73ff6e4fdf46199f6aa3a04b72a34a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:52.187718 22068 raft_consensus.cc:3054] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.192394 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c73ff6e4fdf46199f6aa3a04b72a34a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:52.193161 22213 raft_consensus.cc:3054] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.195137 22461 tablet_bootstrap.cc:654] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.197947 21715 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_14794d2519464ce3bb6d9c0dc8f7758b" } modify_external_catalogs: true
I20250112 08:56:52.198599 21715 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_14794d2519464ce3bb6d9c0dc8f7758b" } modify_external_catalogs: true
W20250112 08:56:52.205176 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e43f8e3061c54472a80f3dad3d2b8d86 has no consensus state
I20250112 08:56:52.218050 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]). Expected version 2 got 0
I20250112 08:56:52.221567 22461 tablet_bootstrap.cc:492] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:52.222118 22461 ts_tablet_manager.cc:1397] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.001s
I20250112 08:56:52.222041 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]). Expected version 2 got 0
I20250112 08:56:52.222702 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:52.224761 22336 tablet_service.cc:1467] Processing CreateTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e (DEFAULT_TABLE table=default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.225939 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e63fab9679415ea2dfb4a032b7d80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.226034 22048 tablet_service.cc:1467] Processing CreateTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e (DEFAULT_TABLE table=default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:52.226440 21711 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e23582f78e674fd8b274669bc1e3a58a consensus state has no leader
I20250112 08:56:52.227108 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e63fab9679415ea2dfb4a032b7d80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.229473 22192 tablet_service.cc:1467] Processing CreateTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e (DEFAULT_TABLE table=default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.230192 21714 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]). Expected version 2 got 0
I20250112 08:56:52.230573 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9e63fab9679415ea2dfb4a032b7d80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.237187 22461 raft_consensus.cc:357] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.244042 22461 raft_consensus.cc:383] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.245110 22461 raft_consensus.cc:738] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.246523 22461 consensus_queue.cc:260] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.256112 22461 ts_tablet_manager.cc:1428] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.033s user 0.005s sys 0.002s
I20250112 08:56:52.257490 22461 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:52.262441 22213 raft_consensus.cc:2463] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:52.263428 22068 raft_consensus.cc:2463] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:52.264636 22477 ts_tablet_manager.cc:1918] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.266018 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 48b63eff49fc4b829afc8160b266fee5 (table default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]) successfully deleted
I20250112 08:56:52.267294 22456 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.267719 22456 ts_tablet_manager.cc:1397] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.101s user 0.023s sys 0.023s
I20250112 08:56:52.270635 22458 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.270314 22456 raft_consensus.cc:357] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.271024 22456 raft_consensus.cc:383] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.271315 22456 raft_consensus.cc:738] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.271966 22456 consensus_queue.cc:260] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.274257 22456 ts_tablet_manager.cc:1428] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 08:56:52.275179 22456 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:52.276752 22458 tablet_bootstrap.cc:654] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.263621 22290 leader_election.cc:304] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:52.281489 22469 raft_consensus.cc:2798] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:52.281989 22469 raft_consensus.cc:695] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:56:52.282874 22469 consensus_queue.cc:237] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.286960 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:52.289680 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 2c73ff6e4fdf46199f6aa3a04b72a34a
I20250112 08:56:52.300371 22456 tablet_bootstrap.cc:654] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.301301 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.302050 22475 tablet_replica.cc:331] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:52.302755 22475 raft_consensus.cc:2238] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.303028 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:52.304092 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:52.303663 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:52.303301 22475 raft_consensus.cc:2267] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.304904 22476 tablet_replica.cc:331] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:52.306852 22475 ts_tablet_manager.cc:1905] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.308157 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:52.313894 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b63eff49fc4b829afc8160b266fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.322005 22476 raft_consensus.cc:2238] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:52.326375 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_07538585007d492c837de9799ced16c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:52.326218 22464 raft_consensus.cc:491] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:56:52.328193 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07538585007d492c837de9799ced16c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.328197 22464 raft_consensus.cc:513] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.334218 22476 pending_rounds.cc:62] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:56:52.334627 22476 pending_rounds.cc:69] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409381525590016 op_type: NO_OP noop_request { }
I20250112 08:56:52.335175 22476 raft_consensus.cc:2883] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:56:52.335687 22476 raft_consensus.cc:2267] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.341531 22476 ts_tablet_manager.cc:1905] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.343726 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.347204 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_3aba77f931194c9fbb79efbb1a3be03c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:52.348985 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aba77f931194c9fbb79efbb1a3be03c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.353036 22464 leader_election.cc:290] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:52.353662 22464 raft_consensus.cc:491] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:52.353478 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_4cb2fecb0a54421382f3ffa9c09e393c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:52.354133 22464 raft_consensus.cc:513] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:56:52.355178 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cb2fecb0a54421382f3ffa9c09e393c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.357580 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23582f78e674fd8b274669bc1e3a58a" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:52.358361 22213 raft_consensus.cc:2463] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:52.361393 21999 leader_election.cc:304] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:52.365031 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:52.368571 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c934e1a62444e64911a279f6a544c96" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:52.369756 22213 raft_consensus.cc:2463] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:52.370019 22477 log.cc:1198] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:52.371097 21999 leader_election.cc:304] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:52.375216 22476 ts_tablet_manager.cc:1918] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:52.375873 22476 log.cc:1198] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/2c73ff6e4fdf46199f6aa3a04b72a34a
I20250112 08:56:52.371062 22477 ts_tablet_manager.cc:1939] T 48b63eff49fc4b829afc8160b266fee5 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:52.379781 22476 ts_tablet_manager.cc:1939] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:52.381727 22477 tablet_replica.cc:331] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:52.382478 22477 raft_consensus.cc:2238] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.382884 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]) successfully deleted
I20250112 08:56:52.382937 22477 raft_consensus.cc:2267] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.387646 22477 ts_tablet_manager.cc:1905] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.390710 22461 tablet_bootstrap.cc:654] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.395411 22458 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:52.385211 22464 leader_election.cc:290] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:52.396373 22464 raft_consensus.cc:2798] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:52.396703 22464 raft_consensus.cc:491] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:52.397028 22464 raft_consensus.cc:3054] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:56:52.401455 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet b9e63fab9679415ea2dfb4a032b7d80e because the tablet was not found. No further retry: Not found: Tablet not found: b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:52.403756 22464 raft_consensus.cc:513] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.413743 22493 raft_consensus.cc:2798] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:52.414146 22493 raft_consensus.cc:491] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:52.414438 22493 raft_consensus.cc:3054] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.416579 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 48b63eff49fc4b829afc8160b266fee5 (table default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]) successfully deleted
W20250112 08:56:52.418241 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 47226.5us
I20250112 08:56:52.418959 22458 ts_tablet_manager.cc:1397] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.148s user 0.011s sys 0.001s
I20250112 08:56:52.421875 21717 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:52.423693 22458 raft_consensus.cc:357] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.424851 22458 raft_consensus.cc:383] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.425284 22458 raft_consensus.cc:738] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.427173 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 5157642d667d49aaad41df56be38801f (DEFAULT_TABLE table=default.Table_0f3421a70fde4a968a22dcb2a2db7693 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.428584 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157642d667d49aaad41df56be38801f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.431197 22456 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.431624 22456 ts_tablet_manager.cc:1397] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.157s user 0.026s sys 0.017s
I20250112 08:56:52.434415 22456 raft_consensus.cc:357] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.435113 22456 raft_consensus.cc:383] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.435356 22456 raft_consensus.cc:738] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.441840 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 5157642d667d49aaad41df56be38801f (DEFAULT_TABLE table=default.Table_0f3421a70fde4a968a22dcb2a2db7693 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.441818 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 (DEFAULT_TABLE table=default.Table_e03e9ba8aa6143669b27bd0f94949736 [id=8bb37e8f837d4c5299debd19da8416b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.442965 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157642d667d49aaad41df56be38801f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.442967 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffbab945abd4fa58259a65e4fadd622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.447562 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_e03e9ba8aa6143669b27bd0f94949736" } new_table_name: "default.Table_abb85608f59e40c1bcb4004eb5783cb6" modify_external_catalogs: true
I20250112 08:56:52.447714 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0f3421a70fde4a968a22dcb2a2db7693" } new_table_name: "default.Table_84ad968b22304d95ae866414672e3eb5" modify_external_catalogs: true
I20250112 08:56:52.448930 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_d236ceb20a6c4863b95e6fe50c2c12b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:52.450610 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d236ceb20a6c4863b95e6fe50c2c12b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.457028 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:52.457844 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e23582f78e674fd8b274669bc1e3a58a (table default.Table_b66f0362eec34c34a4efe32381f0d747 [id=1bbafad80c2f4de18948b31d817955c6]). Expected version 1 got 0
I20250112 08:56:52.460422 22458 consensus_queue.cc:260] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.469265 22456 consensus_queue.cc:260] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.471578 22456 ts_tablet_manager.cc:1428] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250112 08:56:52.472666 22456 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:52.479813 22191 tablet_service.cc:1467] Processing CreateTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 (DEFAULT_TABLE table=default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.481128 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43f8e3061c54472a80f3dad3d2b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.485281 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.489780 22458 ts_tablet_manager.cc:1428] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.071s user 0.008s sys 0.000s
I20250112 08:56:52.506033 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7e088c3591104573aef358a41173f181" } modify_external_catalogs: true
I20250112 08:56:52.506254 21709 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b66f0362eec34c34a4efe32381f0d747" } modify_external_catalogs: true
I20250112 08:56:52.506652 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7e088c3591104573aef358a41173f181" } modify_external_catalogs: true
I20250112 08:56:52.506744 21709 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b66f0362eec34c34a4efe32381f0d747" } modify_external_catalogs: true
I20250112 08:56:52.467005 22493 raft_consensus.cc:513] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.516573 22493 leader_election.cc:290] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:52.521324 22464 leader_election.cc:290] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:52.529201 22047 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 (DEFAULT_TABLE table=default.Table_e03e9ba8aa6143669b27bd0f94949736 [id=8bb37e8f837d4c5299debd19da8416b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.530325 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffbab945abd4fa58259a65e4fadd622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.530259 22469 raft_consensus.cc:491] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:52.533416 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 745d1a12356244fa811d4b0434ad1295 (DEFAULT_TABLE table=default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.535099 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745d1a12356244fa811d4b0434ad1295. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:52.537743 21702 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e43f8e3061c54472a80f3dad3d2b8d86 consensus state has no leader
I20250112 08:56:52.541236 22469 raft_consensus.cc:513] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:52.542179 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b63eff49fc4b829afc8160b266fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:52.543068 22476 tablet_replica.cc:331] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:52.543752 22476 raft_consensus.cc:2238] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.544277 22476 raft_consensus.cc:2267] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.544706 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:52.546419 22046 tablet_service.cc:1467] Processing CreateTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 (DEFAULT_TABLE table=default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.547847 22045 tablet_service.cc:1467] Processing CreateTablet for tablet 745d1a12356244fa811d4b0434ad1295 (DEFAULT_TABLE table=default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.549748 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d (DEFAULT_TABLE table=default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.553748 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 (DEFAULT_TABLE table=default.Table_e03e9ba8aa6143669b27bd0f94949736 [id=8bb37e8f837d4c5299debd19da8416b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.554828 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 5157642d667d49aaad41df56be38801f (DEFAULT_TABLE table=default.Table_0f3421a70fde4a968a22dcb2a2db7693 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.555871 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d (DEFAULT_TABLE table=default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.560307 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c934e1a62444e64911a279f6a544c96" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:52.560891 22213 raft_consensus.cc:3054] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.562376 22456 tablet_bootstrap.cc:654] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.566524 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43f8e3061c54472a80f3dad3d2b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.552255 22335 tablet_service.cc:1467] Processing CreateTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 (DEFAULT_TABLE table=default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.568017 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffbab945abd4fa58259a65e4fadd622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.572989 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85059ac5ccd844e3893d214ef9ce6b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.575860 22458 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.577040 22477 ts_tablet_manager.cc:1918] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.577399 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157642d667d49aaad41df56be38801f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.577447 22477 log.cc:1198] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/2c73ff6e4fdf46199f6aa3a04b72a34a
I20250112 08:56:52.578102 22477 ts_tablet_manager.cc:1939] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:52.580724 22477 tablet_replica.cc:331] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:52.581254 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e43f8e3061c54472a80f3dad3d2b8d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.581486 22477 raft_consensus.cc:2238] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.581931 22477 raft_consensus.cc:2267] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.583678 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745d1a12356244fa811d4b0434ad1295. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.585551 22477 ts_tablet_manager.cc:1905] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.561293 22469 leader_election.cc:290] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:52.592726 22476 ts_tablet_manager.cc:1905] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.596410 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85059ac5ccd844e3893d214ef9ce6b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.599346 22464 raft_consensus.cc:491] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:52.600419 22477 ts_tablet_manager.cc:1918] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.600834 22477 log.cc:1198] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:52.600148 22464 raft_consensus.cc:513] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.601488 22477 ts_tablet_manager.cc:1939] T b9e63fab9679415ea2dfb4a032b7d80e P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:52.603953 22477 tablet_replica.cc:331] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:52.604591 22464 leader_election.cc:290] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:52.616567 22461 tablet_bootstrap.cc:492] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:52.617020 22461 ts_tablet_manager.cc:1397] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.360s user 0.006s sys 0.012s
I20250112 08:56:52.619364 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_78050ca964624292a8e2a3c2a10d59af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:52.620451 22290 leader_election.cc:343] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 48b63eff49fc4b829afc8160b266fee5
W20250112 08:56:52.621167 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78050ca964624292a8e2a3c2a10d59af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.623687 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_cd341c9b848147888660098e58a0499e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:52.625051 22458 tablet_bootstrap.cc:654] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:52.625473 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd341c9b848147888660098e58a0499e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.604781 22477 raft_consensus.cc:2238] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.628463 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]) successfully deleted
I20250112 08:56:52.629086 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet b9e63fab9679415ea2dfb4a032b7d80e (table default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]) successfully deleted
I20250112 08:56:52.629076 22477 raft_consensus.cc:2267] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.612222 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23582f78e674fd8b274669bc1e3a58a" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:52.633056 22475 ts_tablet_manager.cc:1918] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.633479 22475 log.cc:1198] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/2c73ff6e4fdf46199f6aa3a04b72a34a
I20250112 08:56:52.634111 22475 ts_tablet_manager.cc:1939] T 2c73ff6e4fdf46199f6aa3a04b72a34a P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:52.634835 22213 raft_consensus.cc:2463] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:52.641602 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 745d1a12356244fa811d4b0434ad1295 (DEFAULT_TABLE table=default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:52.643150 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745d1a12356244fa811d4b0434ad1295. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.645324 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23582f78e674fd8b274669bc1e3a58a" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:52.647187 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c934e1a62444e64911a279f6a544c96" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:52.647823 22357 raft_consensus.cc:2463] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:52.649292 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c934e1a62444e64911a279f6a544c96" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:52.649766 22357 raft_consensus.cc:3054] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.652055 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d (DEFAULT_TABLE table=default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:52.652398 22290 leader_election.cc:343] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:52.652803 22290 leader_election.cc:304] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
W20250112 08:56:52.653532 22290 raft_consensus.cc:2681] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:56:52.652323 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e84f97537c74fb1ba32f7861bf27457" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:52.654145 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85059ac5ccd844e3893d214ef9ce6b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:52.652753 22212 raft_consensus.cc:3054] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:52.659330 22357 raft_consensus.cc:2463] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:52.655405 22470 raft_consensus.cc:491] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:52.664947 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23582f78e674fd8b274669bc1e3a58a" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:52.667394 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e84f97537c74fb1ba32f7861bf27457" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:52.619738 22461 raft_consensus.cc:357] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.669222 22461 raft_consensus.cc:383] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.669500 22461 raft_consensus.cc:738] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.670203 22461 consensus_queue.cc:260] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.671213 22358 raft_consensus.cc:2463] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:52.672554 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 2c73ff6e4fdf46199f6aa3a04b72a34a (table default.Table_14794d2519464ce3bb6d9c0dc8f7758b [id=dbe51ac33754416f84c887ba5a2af3d6]) successfully deleted
I20250112 08:56:52.674889 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1eb512e21fe24840b674e404ed647af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:52.677146 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eb512e21fe24840b674e404ed647af7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:52.678028 22470 raft_consensus.cc:513] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.680701 22477 ts_tablet_manager.cc:1905] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.680753 21999 leader_election.cc:304] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
W20250112 08:56:52.693364 22002 leader_election.cc:343] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:52.694027 22213 raft_consensus.cc:2463] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:52.694077 22464 raft_consensus.cc:2798] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:52.708582 22461 ts_tablet_manager.cc:1428] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.091s user 0.003s sys 0.004s
I20250112 08:56:52.718400 22464 raft_consensus.cc:695] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:52.720850 22464 consensus_queue.cc:237] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.729410 22461 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:52.687845 22470 leader_election.cc:290] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:56:52.748189 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 48b63eff49fc4b829afc8160b266fee5 because the tablet was not found. No further retry: Not found: Tablet not found: 48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:52.748929 22475 tablet_replica.cc:331] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: stopping tablet replica
W20250112 08:56:52.749264 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet b9e63fab9679415ea2dfb4a032b7d80e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9e63fab9679415ea2dfb4a032b7d80e already in progress: creating tablet
I20250112 08:56:52.749699 22475 raft_consensus.cc:2238] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.750188 22475 raft_consensus.cc:2267] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.753425 22458 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:52.753767 22458 ts_tablet_manager.cc:1397] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.178s user 0.006s sys 0.014s
I20250112 08:56:52.756258 21999 leader_election.cc:304] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:52.756484 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e84f97537c74fb1ba32f7861bf27457" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:52.756505 22458 raft_consensus.cc:357] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:52.757021 21999 raft_consensus.cc:2681] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 08:56:52.758067 22149 leader_election.cc:343] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:52.763408 22476 ts_tablet_manager.cc:1918] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.765938 21713 catalog_manager.cc:5526] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:52.767829 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:52.755784 22212 raft_consensus.cc:2463] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:52.771999 22476 log.cc:1198] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/48b63eff49fc4b829afc8160b266fee5
I20250112 08:56:52.772720 22476 ts_tablet_manager.cc:1939] T 48b63eff49fc4b829afc8160b266fee5 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:52.772946 21999 leader_election.cc:304] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:52.773689 22464 raft_consensus.cc:2798] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:52.757160 22458 raft_consensus.cc:383] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.774121 22464 raft_consensus.cc:695] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:52.774102 22458 raft_consensus.cc:738] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.775213 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 48b63eff49fc4b829afc8160b266fee5 (table default.Table_144c1527f47e4533848766fc39213caa [id=384bd09cd5264829a09783af5539802a]) successfully deleted
I20250112 08:56:52.775161 22458 consensus_queue.cc:260] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.775136 22464 consensus_queue.cc:237] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:56:52.777511 22458 ts_tablet_manager.cc:1428] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.023s user 0.002s sys 0.004s
I20250112 08:56:52.778892 22458 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.786844 22458 tablet_bootstrap.cc:654] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.792738 22475 ts_tablet_manager.cc:1905] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.799060 22461 tablet_bootstrap.cc:654] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.809211 22456 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.809494 22458 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:52.809801 22456 ts_tablet_manager.cc:1397] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.337s user 0.010s sys 0.013s
I20250112 08:56:52.809854 22458 ts_tablet_manager.cc:1397] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.003s
I20250112 08:56:52.812011 22458 raft_consensus.cc:357] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.812654 22458 raft_consensus.cc:383] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.812999 22458 raft_consensus.cc:738] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.814950 22458 consensus_queue.cc:260] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.815704 22477 ts_tablet_manager.cc:1918] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.812419 22456 raft_consensus.cc:357] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.816792 22477 log.cc:1198] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/6e84f97537c74fb1ba32f7861bf27457
I20250112 08:56:52.817124 22456 raft_consensus.cc:383] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.824453 22477 ts_tablet_manager.cc:1939] T 6e84f97537c74fb1ba32f7861bf27457 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:52.824806 22456 raft_consensus.cc:738] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.828526 22456 consensus_queue.cc:260] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.825990 22464 raft_consensus.cc:491] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:56:52.829787 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5157642d667d49aaad41df56be38801f consensus state has no leader
I20250112 08:56:52.829550 22464 raft_consensus.cc:513] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.831387 22464 leader_election.cc:290] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
W20250112 08:56:52.832376 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ffbab945abd4fa58259a65e4fadd622 consensus state has no leader
I20250112 08:56:52.827066 22458 ts_tablet_manager.cc:1428] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20250112 08:56:52.828533 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e84f97537c74fb1ba32f7861bf27457" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:52.837531 22146 leader_election.cc:343] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250112 08:56:52.837976 22456 ts_tablet_manager.cc:1428] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250112 08:56:52.838147 22146 leader_election.cc:304] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:52.838869 22456 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:56:52.839262 22146 raft_consensus.cc:2681] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:56:52.833299 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
W20250112 08:56:52.841285 21999 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:52.842741 22458 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.848879 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 6e84f97537c74fb1ba32f7861bf27457 (table default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]) successfully deleted
I20250112 08:56:52.852108 22456 tablet_bootstrap.cc:654] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.856597 22475 ts_tablet_manager.cc:1918] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.862176 22475 log.cc:1198] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/6e84f97537c74fb1ba32f7861bf27457
I20250112 08:56:52.862764 22458 tablet_bootstrap.cc:654] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.863147 22475 ts_tablet_manager.cc:1939] T 6e84f97537c74fb1ba32f7861bf27457 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:52.868077 22456 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.868523 22456 ts_tablet_manager.cc:1397] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.009s
I20250112 08:56:52.868461 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 6e84f97537c74fb1ba32f7861bf27457 (table default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]) successfully deleted
I20250112 08:56:52.870783 22456 raft_consensus.cc:357] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.871560 22456 raft_consensus.cc:383] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.880982 22458 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:52.881425 22456 raft_consensus.cc:738] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.881590 22458 ts_tablet_manager.cc:1397] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.006s
I20250112 08:56:52.883445 22456 consensus_queue.cc:260] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.889212 22458 raft_consensus.cc:357] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.890228 22458 raft_consensus.cc:383] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.890583 22458 raft_consensus.cc:738] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.891189 22456 ts_tablet_manager.cc:1428] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250112 08:56:52.893296 22456 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:52.904345 22458 consensus_queue.cc:260] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.898963 22456 tablet_bootstrap.cc:654] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:52.908640 22458 ts_tablet_manager.cc:1428] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.027s user 0.009s sys 0.001s
I20250112 08:56:52.910651 22458 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.920746 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 745d1a12356244fa811d4b0434ad1295
I20250112 08:56:52.925148 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:52.926601 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.928412 22458 tablet_bootstrap.cc:654] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:52.932344 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 745d1a12356244fa811d4b0434ad1295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 745d1a12356244fa811d4b0434ad1295 already in progress: creating tablet
W20250112 08:56:52.933135 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 745d1a12356244fa811d4b0434ad1295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 745d1a12356244fa811d4b0434ad1295 already in progress: creating tablet
I20250112 08:56:52.937213 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet e23582f78e674fd8b274669bc1e3a58a
I20250112 08:56:52.940407 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet e23582f78e674fd8b274669bc1e3a58a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:52.941362 22477 tablet_replica.cc:331] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:52.942119 22477 raft_consensus.cc:2238] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:52.942889 22477 pending_rounds.cc:62] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Trying to abort 1 pending ops.
I20250112 08:56:52.943163 22477 pending_rounds.cc:69] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409383535603712 op_type: NO_OP noop_request { }
I20250112 08:56:52.943624 22477 raft_consensus.cc:2883] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:56:52.944041 22477 raft_consensus.cc:2267] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.946419 22477 ts_tablet_manager.cc:1905] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.947057 22458 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:52.947430 22458 ts_tablet_manager.cc:1397] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.003s
I20250112 08:56:52.949590 22458 raft_consensus.cc:357] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.949928 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet e23582f78e674fd8b274669bc1e3a58a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:52.950201 22458 raft_consensus.cc:383] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.950444 22458 raft_consensus.cc:738] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.950631 22475 tablet_replica.cc:331] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:52.951399 22475 raft_consensus.cc:2238] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:52.951329 22458 consensus_queue.cc:260] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.951877 22475 raft_consensus.cc:2267] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:52.954006 22475 ts_tablet_manager.cc:1905] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:52.957639 22456 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:52.957674 22458 ts_tablet_manager.cc:1428] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20250112 08:56:52.961551 22456 ts_tablet_manager.cc:1397] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.008s
I20250112 08:56:52.964301 22458 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:52.968276 22456 raft_consensus.cc:357] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.969017 22456 raft_consensus.cc:383] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:52.969303 22456 raft_consensus.cc:738] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:52.969949 22456 consensus_queue.cc:260] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:52.976539 22477 ts_tablet_manager.cc:1918] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:52.976958 22477 log.cc:1198] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/e23582f78e674fd8b274669bc1e3a58a
I20250112 08:56:52.977715 22477 ts_tablet_manager.cc:1939] T e23582f78e674fd8b274669bc1e3a58a P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
W20250112 08:56:52.979635 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 2c73ff6e4fdf46199f6aa3a04b72a34a because the tablet was not found. No further retry: Not found: Tablet not found: 2c73ff6e4fdf46199f6aa3a04b72a34a
I20250112 08:56:52.983563 22456 ts_tablet_manager.cc:1428] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250112 08:56:52.876343 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:52.925980 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:52.986119 22002 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:52.986542 22002 leader_election.cc:304] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:52.987656 22496 raft_consensus.cc:2743] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:52.988678 22456 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:52.991428 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet e23582f78e674fd8b274669bc1e3a58a (table default.Table_b66f0362eec34c34a4efe32381f0d747 [id=1bbafad80c2f4de18948b31d817955c6]) successfully deleted
I20250112 08:56:52.998452 22475 ts_tablet_manager.cc:1918] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:52.951521 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet e23582f78e674fd8b274669bc1e3a58a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:53.002071 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 745d1a12356244fa811d4b0434ad1295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 745d1a12356244fa811d4b0434ad1295 already in progress: creating tablet
I20250112 08:56:53.005223 22475 log.cc:1198] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/e23582f78e674fd8b274669bc1e3a58a
I20250112 08:56:53.006012 22475 ts_tablet_manager.cc:1939] T e23582f78e674fd8b274669bc1e3a58a P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:53.004904 22456 tablet_bootstrap.cc:654] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.009881 22458 tablet_bootstrap.cc:654] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.012485 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet e23582f78e674fd8b274669bc1e3a58a (table default.Table_b66f0362eec34c34a4efe32381f0d747 [id=1bbafad80c2f4de18948b31d817955c6]) successfully deleted
I20250112 08:56:53.017869 22456 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.018275 22456 ts_tablet_manager.cc:1397] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.005s
I20250112 08:56:53.024485 22476 tablet_replica.cc:331] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:53.025902 22476 raft_consensus.cc:2238] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.025640 22456 raft_consensus.cc:357] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.026360 22456 raft_consensus.cc:383] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.026628 22456 raft_consensus.cc:738] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.030762 22476 raft_consensus.cc:2267] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.031751 22456 consensus_queue.cc:260] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.034186 22476 ts_tablet_manager.cc:1905] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:53.034526 22458 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:53.034960 22458 ts_tablet_manager.cc:1397] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.008s
I20250112 08:56:53.038120 22458 raft_consensus.cc:357] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.038700 22458 raft_consensus.cc:383] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.038879 22458 raft_consensus.cc:738] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.039793 22458 consensus_queue.cc:260] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.049319 22458 ts_tablet_manager.cc:1428] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250112 08:56:53.059114 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:53.060940 22456 ts_tablet_manager.cc:1428] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20250112 08:56:53.061419 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_07538585007d492c837de9799ced16c7" } new_table_name: "default.Table_d090a907117f41cc97cc09e0114998be" modify_external_catalogs: true
I20250112 08:56:53.062041 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cd341c9b848147888660098e58a0499e" } new_table_name: "default.Table_a6bd091b79fd4a2c952ed2f0f95da386" modify_external_catalogs: true
I20250112 08:56:53.064167 22456 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:53.065290 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_dba029194bb34f3ba82f24b9083db1b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.067523 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dba029194bb34f3ba82f24b9083db1b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.069830 22456 tablet_bootstrap.cc:654] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.072873 22475 tablet_replica.cc:331] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:53.073645 22475 raft_consensus.cc:2238] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.074088 22475 raft_consensus.cc:2267] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.083652 22475 ts_tablet_manager.cc:1905] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:53.104089 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_7fbc84cb322f46eab0f1ccf83944405c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.106056 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fbc84cb322f46eab0f1ccf83944405c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.113380 22475 ts_tablet_manager.cc:1918] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.114075 22475 log.cc:1198] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:53.115141 22475 ts_tablet_manager.cc:1939] T b9e63fab9679415ea2dfb4a032b7d80e P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:53.118919 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet b9e63fab9679415ea2dfb4a032b7d80e (table default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]) successfully deleted
W20250112 08:56:53.121299 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa641fe365844c9eaa6ffcbf548d82f7 has no consensus state
I20250112 08:56:53.138054 22461 tablet_bootstrap.cc:492] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:53.138010 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_fad3e6f7b8f84d748b8429927a55070a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.140027 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fad3e6f7b8f84d748b8429927a55070a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.140540 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_78050ca964624292a8e2a3c2a10d59af" } modify_external_catalogs: true
I20250112 08:56:53.142493 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_78050ca964624292a8e2a3c2a10d59af" } modify_external_catalogs: true
I20250112 08:56:53.142793 21711 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d236ceb20a6c4863b95e6fe50c2c12b0" } modify_external_catalogs: true
I20250112 08:56:53.143350 21711 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d236ceb20a6c4863b95e6fe50c2c12b0" } modify_external_catalogs: true
I20250112 08:56:53.138998 22461 ts_tablet_manager.cc:1397] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.410s user 0.005s sys 0.007s
I20250112 08:56:53.146818 22461 raft_consensus.cc:357] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.147538 22461 raft_consensus.cc:383] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.147819 22461 raft_consensus.cc:738] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:56:53.160346 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48adce00f3644192880a8b16b5e514d9 has no consensus state
I20250112 08:56:53.167469 22461 consensus_queue.cc:260] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.173477 22461 ts_tablet_manager.cc:1428] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250112 08:56:53.174425 22461 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:53.177809 22456 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.178298 22456 ts_tablet_manager.cc:1397] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.114s user 0.014s sys 0.029s
I20250112 08:56:53.180629 22456 raft_consensus.cc:357] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.181269 22456 raft_consensus.cc:383] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.182683 22456 raft_consensus.cc:738] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.183511 22456 consensus_queue.cc:260] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.190196 22456 ts_tablet_manager.cc:1428] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250112 08:56:53.193403 22456 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:53.200258 22470 raft_consensus.cc:491] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.200881 22470 raft_consensus.cc:513] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.204762 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffbab945abd4fa58259a65e4fadd622" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:53.205403 22068 raft_consensus.cc:2463] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:53.206622 22146 leader_election.cc:304] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:53.204224 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffbab945abd4fa58259a65e4fadd622" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.209636 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
I20250112 08:56:53.210486 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 1 got 0
W20250112 08:56:53.211614 22149 leader_election.cc:343] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.206543 22332 tablet_service.cc:1467] Processing CreateTablet for tablet f152b58bd9114691bab99c23b51a5e89 (DEFAULT_TABLE table=default.Table_4cb2fecb0a54421382f3ffa9c09e393c [id=6ae86a04705444a6ba14742d84959e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.203552 22470 leader_election.cc:290] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.213254 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f152b58bd9114691bab99c23b51a5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.213594 22470 raft_consensus.cc:2798] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:53.214075 22470 raft_consensus.cc:491] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:53.214511 22470 raft_consensus.cc:3054] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.217025 22336 tablet_service.cc:1467] Processing CreateTablet for tablet e456834c46414ae9889e8ff088b79c29 (DEFAULT_TABLE table=default.Table_3aba77f931194c9fbb79efbb1a3be03c [id=70b6c08dd74f4211ad7034a109e70eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.218148 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e456834c46414ae9889e8ff088b79c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.211548 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_84ad968b22304d95ae866414672e3eb5 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 1 got 0
I20250112 08:56:53.221693 22496 raft_consensus.cc:491] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.222496 22045 tablet_service.cc:1467] Processing CreateTablet for tablet f152b58bd9114691bab99c23b51a5e89 (DEFAULT_TABLE table=default.Table_4cb2fecb0a54421382f3ffa9c09e393c [id=6ae86a04705444a6ba14742d84959e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.223341 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
I20250112 08:56:53.223613 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f152b58bd9114691bab99c23b51a5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.223690 22044 tablet_service.cc:1467] Processing CreateTablet for tablet e456834c46414ae9889e8ff088b79c29 (DEFAULT_TABLE table=default.Table_3aba77f931194c9fbb79efbb1a3be03c [id=70b6c08dd74f4211ad7034a109e70eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.224016 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 1 got 0
I20250112 08:56:53.225155 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_84ad968b22304d95ae866414672e3eb5 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 1 got 0
I20250112 08:56:53.225653 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 48adce00f3644192880a8b16b5e514d9 (DEFAULT_TABLE table=default.Table_d090a907117f41cc97cc09e0114998be [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.225795 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 48adce00f3644192880a8b16b5e514d9 (DEFAULT_TABLE table=default.Table_d090a907117f41cc97cc09e0114998be [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.226631 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adce00f3644192880a8b16b5e514d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.222180 22496 raft_consensus.cc:513] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.229265 22496 leader_election.cc:290] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:53.232030 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d1a12356244fa811d4b0434ad1295" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:53.232228 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745d1a12356244fa811d4b0434ad1295" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.232638 22212 raft_consensus.cc:2463] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:53.233632 21999 leader_election.cc:304] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:53.234752 22190 tablet_service.cc:1467] Processing CreateTablet for tablet f152b58bd9114691bab99c23b51a5e89 (DEFAULT_TABLE table=default.Table_4cb2fecb0a54421382f3ffa9c09e393c [id=6ae86a04705444a6ba14742d84959e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.235787 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f152b58bd9114691bab99c23b51a5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.238677 22335 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e84f97537c74fb1ba32f7861bf27457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:51 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:53.240406 22191 tablet_service.cc:1467] Processing CreateTablet for tablet e456834c46414ae9889e8ff088b79c29 (DEFAULT_TABLE table=default.Table_3aba77f931194c9fbb79efbb1a3be03c [id=70b6c08dd74f4211ad7034a109e70eb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.241444 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e456834c46414ae9889e8ff088b79c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.244639 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 48adce00f3644192880a8b16b5e514d9 (DEFAULT_TABLE table=default.Table_d090a907117f41cc97cc09e0114998be [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.245719 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adce00f3644192880a8b16b5e514d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:53.248775 22002 leader_election.cc:343] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.251282 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:53.252247 22477 tablet_replica.cc:331] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:53.253012 22477 raft_consensus.cc:2238] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.253501 22477 raft_consensus.cc:2267] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.256798 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:53.257658 22477 ts_tablet_manager.cc:1905] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:56:53.258318 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet b9e63fab9679415ea2dfb4a032b7d80e because the tablet was not found. No further retry: Not found: Tablet not found: b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:53.260598 21711 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4c65904a77af4d408f3827c92acfa6a8
I20250112 08:56:53.262537 22496 raft_consensus.cc:491] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.262928 22496 raft_consensus.cc:513] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.264518 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48adce00f3644192880a8b16b5e514d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.265676 22496 leader_election.cc:290] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:53.269757 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
W20250112 08:56:53.271211 21999 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.272006 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e456834c46414ae9889e8ff088b79c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.272280 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:53.273586 22002 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.273972 22002 leader_election.cc:304] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:53.274940 22496 raft_consensus.cc:2743] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:53.276485 22470 raft_consensus.cc:513] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.280140 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffbab945abd4fa58259a65e4fadd622" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:53.281955 22476 ts_tablet_manager.cc:1918] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.282634 22476 log.cc:1198] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/e23582f78e674fd8b274669bc1e3a58a
I20250112 08:56:53.283705 22477 ts_tablet_manager.cc:1918] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.283329 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_cdaf00d3d05147c7b92aabddbb662593"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.284152 22477 log.cc:1198] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/745d1a12356244fa811d4b0434ad1295
I20250112 08:56:53.284778 22476 ts_tablet_manager.cc:1939] T e23582f78e674fd8b274669bc1e3a58a P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
W20250112 08:56:53.285071 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdaf00d3d05147c7b92aabddbb662593 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.284780 22477 ts_tablet_manager.cc:1939] T 745d1a12356244fa811d4b0434ad1295 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:53.290650 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 745d1a12356244fa811d4b0434ad1295 (table default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]) successfully deleted
W20250112 08:56:53.300318 22149 leader_election.cc:343] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.308712 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:53.309921 22475 tablet_replica.cc:331] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:53.311089 22475 raft_consensus.cc:2238] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.311717 22475 raft_consensus.cc:2267] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.315618 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet e23582f78e674fd8b274669bc1e3a58a (table default.Table_b66f0362eec34c34a4efe32381f0d747 [id=1bbafad80c2f4de18948b31d817955c6]) successfully deleted
I20250112 08:56:53.316675 22476 tablet_replica.cc:331] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:53.317412 22476 raft_consensus.cc:2238] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.318043 22476 raft_consensus.cc:2267] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.319181 21708 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 9d3f6073129b4d339fda3bc6ab90f1b6
I20250112 08:56:53.202329 22461 tablet_bootstrap.cc:654] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.321131 22476 ts_tablet_manager.cc:1905] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:53.325343 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_dba029194bb34f3ba82f24b9083db1b6" } new_table_name: "default.Table_e86edc5a626a48a195a3a19d05113984" modify_external_catalogs: true
I20250112 08:56:53.328698 22456 tablet_bootstrap.cc:700] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Time spent opening tablet: real 0.123s user 0.001s sys 0.001s
I20250112 08:56:53.333216 22475 ts_tablet_manager.cc:1905] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:53.332240 22470 leader_election.cc:290] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.342129 22456 tablet_bootstrap.cc:654] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:53.345479 22146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 138951us
I20250112 08:56:53.347240 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffbab945abd4fa58259a65e4fadd622" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:53.347781 22068 raft_consensus.cc:3054] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.350126 22476 ts_tablet_manager.cc:1918] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.350361 22461 tablet_bootstrap.cc:492] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:53.350544 22476 log.cc:1198] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/6e84f97537c74fb1ba32f7861bf27457
I20250112 08:56:53.350811 22461 ts_tablet_manager.cc:1397] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.177s user 0.009s sys 0.001s
I20250112 08:56:53.351344 22476 ts_tablet_manager.cc:1939] T 6e84f97537c74fb1ba32f7861bf27457 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:53.353968 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 6e84f97537c74fb1ba32f7861bf27457 (table default.Table_8e7d87182b7f47f088f746440d3bcc19 [id=ec5e27660a484f30a04bc5acd0b9d8f9]) successfully deleted
W20250112 08:56:53.362356 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10bf912f189a42c19deb744c1c9f4daf has no consensus state
I20250112 08:56:53.371325 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_4cb2fecb0a54421382f3ffa9c09e393c" } new_table_name: "default.Table_b31a7de582934a729d5a6070c4d7fb32" modify_external_catalogs: true
I20250112 08:56:53.377722 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_4165821e9c5947b3bc177a3c85d8ff79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.379823 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4165821e9c5947b3bc177a3c85d8ff79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.384878 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0dc30e0598234d76a083f38a01cfd342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.386888 22461 raft_consensus.cc:357] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.387588 22461 raft_consensus.cc:383] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.388681 22458 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:53.390312 22456 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.390710 22456 ts_tablet_manager.cc:1397] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.197s user 0.004s sys 0.004s
I20250112 08:56:53.392958 22456 raft_consensus.cc:357] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.393729 22456 raft_consensus.cc:383] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.393988 22456 raft_consensus.cc:738] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.394398 22458 tablet_bootstrap.cc:654] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.394696 22456 consensus_queue.cc:260] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:53.403682 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f152b58bd9114691bab99c23b51a5e89 consensus state has no leader
I20250112 08:56:53.412701 22519 raft_consensus.cc:491] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.413720 22519 raft_consensus.cc:513] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.416260 22475 ts_tablet_manager.cc:1918] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.416651 22475 log.cc:1198] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/745d1a12356244fa811d4b0434ad1295
I20250112 08:56:53.416965 22519 leader_election.cc:290] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.417321 22475 ts_tablet_manager.cc:1939] T 745d1a12356244fa811d4b0434ad1295 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:53.418272 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:53.418907 22067 raft_consensus.cc:2463] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:53.418530 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.387883 22461 raft_consensus.cc:738] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.416280 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.420177 22146 leader_election.cc:304] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:53.420245 22461 consensus_queue.cc:260] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:53.420861 22149 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.421101 22519 raft_consensus.cc:2798] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:53.421438 22519 raft_consensus.cc:491] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:53.421747 22519 raft_consensus.cc:3054] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.422489 22456 ts_tablet_manager.cc:1428] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250112 08:56:53.430289 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:53.431831 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet b9e63fab9679415ea2dfb4a032b7d80e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9e63fab9679415ea2dfb4a032b7d80e already in progress: creating tablet
I20250112 08:56:53.432547 22461 ts_tablet_manager.cc:1428] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.081s user 0.006s sys 0.003s
I20250112 08:56:53.433487 22461 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:53.437609 22519 raft_consensus.cc:513] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.445183 22461 tablet_bootstrap.cc:654] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.445288 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 745d1a12356244fa811d4b0434ad1295 (table default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]) successfully deleted
I20250112 08:56:53.443148 22456 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:53.453776 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet b9e63fab9679415ea2dfb4a032b7d80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:53.454578 22476 tablet_replica.cc:331] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:53.455206 22476 raft_consensus.cc:2238] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:53.455626 22476 raft_consensus.cc:2267] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:53.457604 22456 tablet_bootstrap.cc:654] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.459419 22476 ts_tablet_manager.cc:1905] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:56:53.459755 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.469753 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:56:53.471675 22149 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.474061 22519 leader_election.cc:290] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.474270 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:53.474862 22067 raft_consensus.cc:3054] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.479383 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_fad3e6f7b8f84d748b8429927a55070a" } modify_external_catalogs: true
I20250112 08:56:53.480188 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_fad3e6f7b8f84d748b8429927a55070a" } modify_external_catalogs: true
I20250112 08:56:53.482173 22456 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.482565 22456 ts_tablet_manager.cc:1397] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.007s
I20250112 08:56:53.484993 22456 raft_consensus.cc:357] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.485651 22456 raft_consensus.cc:383] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.485919 22456 raft_consensus.cc:738] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.487697 22476 ts_tablet_manager.cc:1918] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:53.492955 22456 consensus_queue.cc:260] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.499783 22068 raft_consensus.cc:2463] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:53.501395 22146 leader_election.cc:304] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:53.502961 22519 raft_consensus.cc:2798] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:53.503019 22067 raft_consensus.cc:2463] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:53.503605 22519 raft_consensus.cc:695] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:53.504725 22519 consensus_queue.cc:237] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.509150 22146 leader_election.cc:304] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:53.510169 22470 raft_consensus.cc:2798] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 08:56:53.392127 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dc30e0598234d76a083f38a01cfd342 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.523855 22476 log.cc:1198] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/b9e63fab9679415ea2dfb4a032b7d80e
I20250112 08:56:53.523764 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cdaf00d3d05147c7b92aabddbb662593" } modify_external_catalogs: true
I20250112 08:56:53.524529 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cdaf00d3d05147c7b92aabddbb662593" } modify_external_catalogs: true
I20250112 08:56:53.529613 22476 ts_tablet_manager.cc:1939] T b9e63fab9679415ea2dfb4a032b7d80e P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:53.531881 22456 ts_tablet_manager.cc:1428] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20250112 08:56:53.532811 22456 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:53.538388 22458 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:53.538780 22458 ts_tablet_manager.cc:1397] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.150s user 0.023s sys 0.015s
I20250112 08:56:53.541128 22458 raft_consensus.cc:357] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.541744 22458 raft_consensus.cc:383] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.542061 22458 raft_consensus.cc:738] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.542691 22458 consensus_queue.cc:260] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.544334 22470 raft_consensus.cc:695] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:53.544755 22458 ts_tablet_manager.cc:1428] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 08:56:53.548583 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.548946 22470 consensus_queue.cc:237] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.557922 22458 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:56:53.565932 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.566479 22461 tablet_bootstrap.cc:492] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:53.566900 22461 ts_tablet_manager.cc:1397] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.134s user 0.017s sys 0.010s
I20250112 08:56:53.569710 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_53ca25b8059f45ffa40dcfc1ed61793c" } new_table_name: "default.Table_fb4954b383d44b93a4cbf55356bda5e4" modify_external_catalogs: true
I20250112 08:56:53.569733 22461 raft_consensus.cc:357] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.570410 22461 raft_consensus.cc:383] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.570652 22461 raft_consensus.cc:738] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:56:53.574919 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 10bf912f189a42c19deb744c1c9f4daf has no consensus state
I20250112 08:56:53.578313 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet b9e63fab9679415ea2dfb4a032b7d80e (table default.Table_771927535b7f4972ae195b3702a277b4 [id=983d3f13535f4181b34e5acd98269ce7]) successfully deleted
I20250112 08:56:53.578461 22461 consensus_queue.cc:260] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.588663 22456 tablet_bootstrap.cc:654] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.596491 21708 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d7cf0c8fb8144ceb8d2f315aba579963
I20250112 08:56:53.599603 22458 tablet_bootstrap.cc:654] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.607369 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7fbc84cb322f46eab0f1ccf83944405c" } new_table_name: "default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3" modify_external_catalogs: true
I20250112 08:56:53.605939 22461 ts_tablet_manager.cc:1428] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.039s user 0.003s sys 0.004s
I20250112 08:56:53.624075 22461 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:53.630544 22461 tablet_bootstrap.cc:654] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.646489 22470 raft_consensus.cc:491] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.646952 22470 raft_consensus.cc:513] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.647574 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf912f189a42c19deb744c1c9f4daf (DEFAULT_TABLE table=default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.648721 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10bf912f189a42c19deb744c1c9f4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.648617 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]). Expected version 1 got 0
I20250112 08:56:53.648978 22470 leader_election.cc:290] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.651307 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f8e3061c54472a80f3dad3d2b8d86" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:53.651541 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f8e3061c54472a80f3dad3d2b8d86" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.651938 22067 raft_consensus.cc:2463] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:53.652184 22357 raft_consensus.cc:2463] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:53.653059 22146 leader_election.cc:304] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:53.654714 22470 raft_consensus.cc:2798] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:53.655061 22470 raft_consensus.cc:491] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:53.655375 22470 raft_consensus.cc:3054] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.656599 21714 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_53ca25b8059f45ffa40dcfc1ed61793c [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 1 got 0
W20250112 08:56:53.659133 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e43f8e3061c54472a80f3dad3d2b8d86 consensus state has no leader
I20250112 08:56:53.666857 21717 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 2 got 0
I20250112 08:56:53.672062 21711 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]). Expected version 1 got 0
I20250112 08:56:53.672747 21711 catalog_manager.cc:5526] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:53.674714 21711 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
W20250112 08:56:53.675503 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fccd34473a6438389cda0b77391c8d7 consensus state has no leader
I20250112 08:56:53.679374 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf912f189a42c19deb744c1c9f4daf (DEFAULT_TABLE table=default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.680385 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10bf912f189a42c19deb744c1c9f4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.682792 22470 raft_consensus.cc:513] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.686085 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf912f189a42c19deb744c1c9f4daf (DEFAULT_TABLE table=default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.687189 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10bf912f189a42c19deb744c1c9f4daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.689445 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (DEFAULT_TABLE table=default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.690501 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.675477 21711 catalog_manager.cc:5526] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:53.691352 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (DEFAULT_TABLE table=default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.691733 21711 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_84ad968b22304d95ae866414672e3eb5 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 1 got 0
I20250112 08:56:53.692381 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.694088 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (DEFAULT_TABLE table=default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.695106 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.707034 22337 tablet_service.cc:1467] Processing CreateTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.708261 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa641fe365844c9eaa6ffcbf548d82f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.706040 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:53.711993 22470 leader_election.cc:290] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.713050 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f8e3061c54472a80f3dad3d2b8d86" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:53.712898 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43f8e3061c54472a80f3dad3d2b8d86" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:53.713649 22067 raft_consensus.cc:3054] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.713732 22357 raft_consensus.cc:3054] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.714877 22335 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:53.716449 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 69a9c2b5a5d044c789a3eda07e8a312d because the tablet was not found. No further retry: Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:53.721154 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:53.721596 22456 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.721990 22456 ts_tablet_manager.cc:1397] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.189s user 0.005s sys 0.006s
I20250112 08:56:53.723647 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:53.723847 22067 raft_consensus.cc:2463] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:53.724112 22357 raft_consensus.cc:2463] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:53.724432 22456 raft_consensus.cc:357] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.725104 22456 raft_consensus.cc:383] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.725374 22146 leader_election.cc:304] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:53.725347 22456 raft_consensus.cc:738] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.726604 22470 raft_consensus.cc:2798] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:53.726418 22456 consensus_queue.cc:260] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.727064 22470 raft_consensus.cc:695] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:53.727969 22470 consensus_queue.cc:237] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.733412 22456 ts_tablet_manager.cc:1428] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250112 08:56:53.734253 22456 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:53.737473 22525 raft_consensus.cc:491] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.737911 22525 raft_consensus.cc:513] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.741182 22525 leader_election.cc:290] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.742920 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152b58bd9114691bab99c23b51a5e89" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.742902 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152b58bd9114691bab99c23b51a5e89" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:53.743800 22067 raft_consensus.cc:2463] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
W20250112 08:56:53.744416 22149 leader_election.cc:343] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:53.726699 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 69a9c2b5a5d044c789a3eda07e8a312d because the tablet was not found. No further retry: Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:53.745268 22146 leader_election.cc:304] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:53.745790 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 69a9c2b5a5d044c789a3eda07e8a312d because the tablet was not found. No further retry: Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:53.746901 22525 raft_consensus.cc:2798] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:53.747424 22525 raft_consensus.cc:491] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:53.747963 22525 raft_consensus.cc:3054] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.748092 22191 tablet_service.cc:1467] Processing CreateTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.749243 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa641fe365844c9eaa6ffcbf548d82f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.748970 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]). Expected version 1 got 0
I20250112 08:56:53.754294 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_bb669f28ccf14e97b75d2d4253cc9ed4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.757287 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 6fccd34473a6438389cda0b77391c8d7 (DEFAULT_TABLE table=default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.757376 22525 raft_consensus.cc:513] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.758392 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fccd34473a6438389cda0b77391c8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.760303 22456 tablet_bootstrap.cc:654] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.761830 22458 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:53.762250 22458 ts_tablet_manager.cc:1397] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.000s
I20250112 08:56:53.762060 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d (DEFAULT_TABLE table=default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.763128 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c2b5a5d044c789a3eda07e8a312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.764621 22458 raft_consensus.cc:357] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.765311 22458 raft_consensus.cc:383] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.765547 22458 raft_consensus.cc:738] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.768483 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152b58bd9114691bab99c23b51a5e89" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:53.769317 22067 raft_consensus.cc:3054] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.773625 22458 consensus_queue.cc:260] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.773958 22525 leader_election.cc:290] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:53.774952 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152b58bd9114691bab99c23b51a5e89" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:56:53.776330 22149 leader_election.cc:343] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.779708 22461 tablet_bootstrap.cc:492] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:53.780123 22461 ts_tablet_manager.cc:1397] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.000s
I20250112 08:56:53.784569 22461 raft_consensus.cc:357] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.785202 22461 raft_consensus.cc:383] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.785368 22461 raft_consensus.cc:738] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.786878 22456 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:53.786926 22458 ts_tablet_manager.cc:1428] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250112 08:56:53.787283 22456 ts_tablet_manager.cc:1397] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.006s
I20250112 08:56:53.787817 22458 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:53.792263 22461 consensus_queue.cc:260] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.794950 22456 raft_consensus.cc:357] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.800890 22067 raft_consensus.cc:2463] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:53.795691 22456 raft_consensus.cc:383] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.801926 22456 raft_consensus.cc:738] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.802189 22146 leader_election.cc:304] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:53.800900 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb669f28ccf14e97b75d2d4253cc9ed4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.802497 22456 consensus_queue.cc:260] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.803067 22461 ts_tablet_manager.cc:1428] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250112 08:56:53.806691 22461 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:53.808363 22519 raft_consensus.cc:2798] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:53.811998 22461 tablet_bootstrap.cc:654] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.821470 22456 ts_tablet_manager.cc:1428] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250112 08:56:53.822104 22496 consensus_queue.cc:1035] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:53.823233 22458 tablet_bootstrap.cc:654] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:53.824019 22519 raft_consensus.cc:695] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:53.824918 22519 consensus_queue.cc:237] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.833830 22044 tablet_service.cc:1467] Processing CreateTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.834976 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa641fe365844c9eaa6ffcbf548d82f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.834681 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 6fccd34473a6438389cda0b77391c8d7 (DEFAULT_TABLE table=default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.835733 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fccd34473a6438389cda0b77391c8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.857134 22461 tablet_bootstrap.cc:492] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:53.857530 22461 ts_tablet_manager.cc:1397] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.000s
I20250112 08:56:53.859292 22045 tablet_service.cc:1467] Processing CreateTablet for tablet 6fccd34473a6438389cda0b77391c8d7 (DEFAULT_TABLE table=default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.859714 22461 raft_consensus.cc:357] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.860368 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fccd34473a6438389cda0b77391c8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.860390 22461 raft_consensus.cc:383] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.860621 22461 raft_consensus.cc:738] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.864462 22458 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:53.864841 22458 ts_tablet_manager.cc:1397] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.003s
I20250112 08:56:53.867066 22458 raft_consensus.cc:357] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.867712 22458 raft_consensus.cc:383] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:53.869091 22458 raft_consensus.cc:738] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:53.869752 22458 consensus_queue.cc:260] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.874136 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d (DEFAULT_TABLE table=default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.874291 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d (DEFAULT_TABLE table=default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.875418 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c2b5a5d044c789a3eda07e8a312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.875228 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c2b5a5d044c789a3eda07e8a312d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:53.892004 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_be70d2c098e64ac4ac3f2fef255d0a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:53.893110 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
W20250112 08:56:53.893774 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be70d2c098e64ac4ac3f2fef255d0a3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.895181 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b31a7de582934a729d5a6070c4d7fb32" } modify_external_catalogs: true
I20250112 08:56:53.895762 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b31a7de582934a729d5a6070c4d7fb32" } modify_external_catalogs: true
I20250112 08:56:53.924960 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_8e031f733adc400f80ef5a314e2c2704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.927515 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e031f733adc400f80ef5a314e2c2704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:53.930455 21704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab -> Peer caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161): Couldn't send request to peer caa33dde8b5248e7ab4d6087d8e9d801. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 08:56:53.937140 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_53a046eb23ec4f73b2e96111509abefe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:53.938733 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53a046eb23ec4f73b2e96111509abefe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:53.951933 22496 raft_consensus.cc:491] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:53.952504 22496 raft_consensus.cc:513] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:53.956251 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48adce00f3644192880a8b16b5e514d9" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:53.956919 22212 raft_consensus.cc:2463] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:53.957697 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48adce00f3644192880a8b16b5e514d9" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:53.958153 21999 leader_election.cc:304] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
W20250112 08:56:53.959092 22002 leader_election.cc:343] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:53.962128 22496 leader_election.cc:290] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:53.965343 22496 raft_consensus.cc:2798] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:53.965373 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_84ad968b22304d95ae866414672e3eb5" } new_table_name: "default.Table_a85a6739b1124d2fb1092ca2bf5ec429" modify_external_catalogs: true
I20250112 08:56:53.969169 22496 raft_consensus.cc:491] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:53.969552 22496 raft_consensus.cc:3054] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:53.974480 21712 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3aba77f931194c9fbb79efbb1a3be03c" } new_table_name: "default.Table_c0a3687f36764454a96695c068067569" modify_external_catalogs: true
I20250112 08:56:53.974710 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54" } modify_external_catalogs: true
I20250112 08:56:53.975189 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54" } modify_external_catalogs: true
I20250112 08:56:53.988160 22458 ts_tablet_manager.cc:1428] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.123s user 0.007s sys 0.000s
I20250112 08:56:53.990895 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
I20250112 08:56:53.991519 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:53.992619 22336 tablet_service.cc:1467] Processing CreateTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 (DEFAULT_TABLE table=default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:53.993734 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0192d66e64d4d39b4bcfd6565dff723. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:54.022408 21748 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc2c0 after lost signal to thread 21685
W20250112 08:56:54.023124 21748 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc580 after lost signal to thread 21751
I20250112 08:56:54.028347 22456 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:54.035454 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:54.037843 22456 tablet_bootstrap.cc:654] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:54.050586 22149 consensus_peers.cc:487] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:54.054059 22456 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.054440 22456 ts_tablet_manager.cc:1397] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.007s
I20250112 08:56:54.056532 22456 raft_consensus.cc:357] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.057173 22456 raft_consensus.cc:383] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.057410 22456 raft_consensus.cc:738] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.059500 22456 consensus_queue.cc:260] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.064414 22190 tablet_service.cc:1467] Processing CreateTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 (DEFAULT_TABLE table=default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.063727 22045 tablet_service.cc:1467] Processing CreateTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 (DEFAULT_TABLE table=default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.065572 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0192d66e64d4d39b4bcfd6565dff723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.066522 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0192d66e64d4d39b4bcfd6565dff723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.073758 22461 consensus_queue.cc:260] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.076409 22458 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:54.076921 22456 ts_tablet_manager.cc:1428] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250112 08:56:54.077767 22456 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:54.095391 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_84ad968b22304d95ae866414672e3eb5 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 1 got 0
I20250112 08:56:54.096956 22519 raft_consensus.cc:491] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.097227 22357 raft_consensus.cc:3054] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.097535 22519 raft_consensus.cc:513] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.101415 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:54.104283 22149 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.105161 22519 leader_election.cc:290] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.105455 22357 raft_consensus.cc:1270] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:54.107542 22535 consensus_queue.cc:1035] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:56:54.109807 22469 raft_consensus.cc:491] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.110234 22469 raft_consensus.cc:513] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.121840 22469 leader_election.cc:290] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.126034 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:54.126035 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5157642d667d49aaad41df56be38801f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:54.126837 22067 raft_consensus.cc:2388] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 53068c99960647728141706e9c50b870 in this term.
I20250112 08:56:54.128324 22290 leader_election.cc:304] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:54.129433 22469 raft_consensus.cc:3054] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.147280 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_d090a907117f41cc97cc09e0114998be [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 1 got 0
I20250112 08:56:54.155831 22456 tablet_bootstrap.cc:654] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.162957 22469 raft_consensus.cc:2743] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:54.183089 22338 tablet_service.cc:1467] Processing CreateTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e (DEFAULT_TABLE table=default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.184352 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f252f0ec641d4bf5b4fe1399a8ced51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.195689 22456 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.196128 22456 ts_tablet_manager.cc:1397] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.002s
I20250112 08:56:54.198793 22456 raft_consensus.cc:357] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.199463 22456 raft_consensus.cc:383] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.199713 22456 raft_consensus.cc:738] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.200433 22456 consensus_queue.cc:260] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.202355 22456 ts_tablet_manager.cc:1428] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250112 08:56:54.168833 21712 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e456834c46414ae9889e8ff088b79c29 consensus state has no leader
W20250112 08:56:54.208528 22533 log.cc:926] Time spent T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20250112 08:56:54.210275 22456 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:54.214939 22458 tablet_bootstrap.cc:654] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.216423 22461 ts_tablet_manager.cc:1428] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.359s user 0.006s sys 0.000s
I20250112 08:56:54.219704 22456 tablet_bootstrap.cc:654] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.220692 22461 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:54.233057 22456 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.228279 22496 raft_consensus.cc:513] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.233997 22456 ts_tablet_manager.cc:1397] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.002s
I20250112 08:56:54.236017 22496 leader_election.cc:290] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:54.236390 22456 raft_consensus.cc:357] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.237169 22456 raft_consensus.cc:383] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.237541 22456 raft_consensus.cc:738] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.238855 22456 consensus_queue.cc:260] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.239557 21709 catalog_manager.cc:5526] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:54.241513 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 2 got 0
I20250112 08:56:54.242246 22456 ts_tablet_manager.cc:1428] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250112 08:56:54.243027 22456 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:54.246601 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:56:54.252758 21714 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet f152b58bd9114691bab99c23b51a5e89
I20250112 08:56:54.257432 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.258651 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 (DEFAULT_TABLE table=default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.259735 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c683c8c64a441e8b5be52b077fdcb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.262061 22335 tablet_service.cc:1467] Processing CreateTablet for tablet d8509fdc7d6b467ab8d445f8a15fa014 (DEFAULT_TABLE table=default.Table_0dc30e0598234d76a083f38a01cfd342 [id=b9ecc50f7e5a454394c47cf7d253ec79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.263394 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8509fdc7d6b467ab8d445f8a15fa014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.264132 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet f152b58bd9114691bab99c23b51a5e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.250319 22190 tablet_service.cc:1467] Processing CreateTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e (DEFAULT_TABLE table=default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.269798 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f252f0ec641d4bf5b4fe1399a8ced51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.254264 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 (DEFAULT_TABLE table=default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.272547 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c683c8c64a441e8b5be52b077fdcb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.257045 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet f152b58bd9114691bab99c23b51a5e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.281577 22470 raft_consensus.cc:491] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.282554 22470 raft_consensus.cc:513] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.284473 22470 leader_election.cc:290] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.285282 22456 tablet_bootstrap.cc:654] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.285627 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:54.290768 22149 leader_election.cc:343] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.255684 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.320104 22543 mvcc.cc:204] Tried to move back new op lower bound from 7113409388890103808 to 7113409386541850624. Current Snapshot: MvccSnapshot[applied={T|T < 7113409388890103808}]
I20250112 08:56:54.340330 22557 tablet_replica.cc:331] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:54.342105 22557 raft_consensus.cc:2238] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:54.343407 22557 pending_rounds.cc:62] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Trying to abort 1 pending ops.
I20250112 08:56:54.343946 22557 pending_rounds.cc:69] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409387838382080 op_type: NO_OP noop_request { }
I20250112 08:56:54.348071 22560 tablet.cc:1721] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:54.348273 22557 raft_consensus.cc:2883] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:56:54.365252 22557 raft_consensus.cc:2267] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:54.369012 22557 ts_tablet_manager.cc:1905] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:54.385775 22461 tablet_bootstrap.cc:654] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.385699 22561 tablet.cc:1721] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:54.392403 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_75e25b72370a4eeaa436c12df91c5fd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.394215 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75e25b72370a4eeaa436c12df91c5fd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:54.395772 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet b0192d66e64d4d39b4bcfd6565dff723 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0192d66e64d4d39b4bcfd6565dff723 already in progress: creating tablet
W20250112 08:56:54.396456 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet f152b58bd9114691bab99c23b51a5e89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f152b58bd9114691bab99c23b51a5e89 already in progress: creating tablet
I20250112 08:56:54.396626 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_39462b870602435cb6f626b7cbf102f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:54.396565 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_002caac6d09040a7bd7484ce1459d559"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.398275 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_002caac6d09040a7bd7484ce1459d559 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.398574 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d090a907117f41cc97cc09e0114998be" } new_table_name: "default.Table_71b1920663e94b938076ab562802e86e" modify_external_catalogs: true
I20250112 08:56:54.399554 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_c3dc458b0d44410898f592553dccc2f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.401264 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3dc458b0d44410898f592553dccc2f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:54.399856 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39462b870602435cb6f626b7cbf102f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.413627 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 2 got 0
I20250112 08:56:54.414533 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_a85a6739b1124d2fb1092ca2bf5ec429 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 2 got 0
I20250112 08:56:54.415321 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:54.416263 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
I20250112 08:56:54.417043 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]). Expected version 1 got 0
I20250112 08:56:54.417635 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]). Expected version 1 got 0
I20250112 08:56:54.420673 21708 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
I20250112 08:56:54.421700 22456 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.422189 22456 ts_tablet_manager.cc:1397] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.179s user 0.047s sys 0.050s
I20250112 08:56:54.424470 22456 raft_consensus.cc:357] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.425151 22456 raft_consensus.cc:383] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.425390 22456 raft_consensus.cc:738] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.425957 22456 consensus_queue.cc:260] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.427855 22456 ts_tablet_manager.cc:1428] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 08:56:54.428619 22456 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:56:54.433560 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48adce00f3644192880a8b16b5e514d9 consensus state has no leader
I20250112 08:56:54.441772 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_71b1920663e94b938076ab562802e86e" } new_table_name: "default.Table_12e188e2f6d24ce68f14d301fdbccf00" modify_external_catalogs: true
I20250112 08:56:54.451929 22357 raft_consensus.cc:1270] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:54.453308 22519 consensus_queue.cc:1035] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:54.453815 21708 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_a85a6739b1124d2fb1092ca2bf5ec429 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 2 got 0
I20250112 08:56:54.457980 22190 tablet_service.cc:1467] Processing CreateTablet for tablet d8509fdc7d6b467ab8d445f8a15fa014 (DEFAULT_TABLE table=default.Table_0dc30e0598234d76a083f38a01cfd342 [id=b9ecc50f7e5a454394c47cf7d253ec79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.459134 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8509fdc7d6b467ab8d445f8a15fa014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.462414 21709 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5" } modify_external_catalogs: true
I20250112 08:56:54.462934 21709 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5" } modify_external_catalogs: true
I20250112 08:56:54.466840 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet f152b58bd9114691bab99c23b51a5e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:54.470428 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48adce00f3644192880a8b16b5e514d9" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:54.481997 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 6fccd34473a6438389cda0b77391c8d7 (table default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]). Expected version 1 got 0
I20250112 08:56:54.483129 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3" } modify_external_catalogs: true
I20250112 08:56:54.485112 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3" } modify_external_catalogs: true
I20250112 08:56:54.486465 22456 tablet_bootstrap.cc:654] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:54.492275 22048 consensus_meta.cc:287] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 0.588s user 0.004s sys 0.003s
I20250112 08:56:54.500289 22456 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.500687 22456 ts_tablet_manager.cc:1397] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.005s
I20250112 08:56:54.503018 22456 raft_consensus.cc:357] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:54.503520 22046 consensus_meta.cc:287] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 0.653s user 0.005s sys 0.003s
I20250112 08:56:54.503728 22456 raft_consensus.cc:383] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.503986 22456 raft_consensus.cc:738] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.505081 22456 consensus_queue.cc:260] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:54.505872 22044 tablet_service.cc:1467] Processing CreateTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e (DEFAULT_TABLE table=default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.507231 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f252f0ec641d4bf5b4fe1399a8ced51e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.507647 22456 ts_tablet_manager.cc:1428] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250112 08:56:54.508463 22456 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:56:54.464233 22001 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.009s sys 0.002s
I20250112 08:56:54.512658 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 (DEFAULT_TABLE table=default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.513782 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c683c8c64a441e8b5be52b077fdcb08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.489607 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:54.521705 22041 tablet_service.cc:1467] Processing CreateTablet for tablet d8509fdc7d6b467ab8d445f8a15fa014 (DEFAULT_TABLE table=default.Table_0dc30e0598234d76a083f38a01cfd342 [id=b9ecc50f7e5a454394c47cf7d253ec79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.522110 22456 tablet_bootstrap.cc:654] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:54.525710 22001 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.574s user 0.010s sys 0.003s
I20250112 08:56:54.526499 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8509fdc7d6b467ab8d445f8a15fa014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.537703 22068 raft_consensus.cc:1270] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:54.541049 22065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:54.554077 22535 consensus_queue.cc:1035] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:56:54.553067 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:54.556579 22066 raft_consensus.cc:2463] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:54.561688 22067 raft_consensus.cc:1270] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:54.562618 22557 ts_tablet_manager.cc:1918] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:54.562990 22557 log.cc:1198] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/f152b58bd9114691bab99c23b51a5e89
I20250112 08:56:54.563805 22557 ts_tablet_manager.cc:1939] T f152b58bd9114691bab99c23b51a5e89 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:54.565595 22064 raft_consensus.cc:1270] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 08:56:54.572093 22002 leader_election.cc:343] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:54.572573 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 51116.2us
W20250112 08:56:54.578054 21704 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_71b1920663e94b938076ab562802e86e [id=711de3e68bc84e8687f455ec32d8259a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48adce00f3644192880a8b16b5e514d9 consensus state has no leader
I20250112 08:56:54.581408 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]) successfully deleted
I20250112 08:56:54.582015 22557 tablet_replica.cc:331] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:54.583614 22557 raft_consensus.cc:2238] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:54.584232 22557 raft_consensus.cc:2267] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:54.586724 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48adce00f3644192880a8b16b5e514d9 consensus state has no leader
I20250112 08:56:54.587554 21708 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7] alter complete (version 1)
W20250112 08:56:54.589166 22146 leader_election.cc:343] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.589673 22146 leader_election.cc:304] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:54.590922 22535 raft_consensus.cc:2743] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:54.595831 22557 ts_tablet_manager.cc:1905] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:54.614915 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:54.615025 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:56:54.626403 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fccd34473a6438389cda0b77391c8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:54.627756 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.628897 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fccd34473a6438389cda0b77391c8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.629602 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:54.632858 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fccd34473a6438389cda0b77391c8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.636792 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_a85a6739b1124d2fb1092ca2bf5ec429" } new_table_name: "default.Table_006c78dcc47545e1b7eb793c44633534" modify_external_catalogs: true
I20250112 08:56:54.640113 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.644203 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_3649588cf8414d9591e8f2ec3fb78e44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:54.641525 22461 tablet_bootstrap.cc:492] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:54.648238 22461 ts_tablet_manager.cc:1397] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.428s user 0.016s sys 0.006s
I20250112 08:56:54.650535 22461 raft_consensus.cc:357] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.651165 22461 raft_consensus.cc:383] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.651409 22461 raft_consensus.cc:738] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:56:54.645731 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 6fccd34473a6438389cda0b77391c8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fccd34473a6438389cda0b77391c8d7 already in progress: creating tablet
W20250112 08:56:54.657225 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 6c683c8c64a441e8b5be52b077fdcb08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c683c8c64a441e8b5be52b077fdcb08 already in progress: creating tablet
I20250112 08:56:54.663026 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:43161
--webserver_interface=127.21.42.252
--webserver_port=36567
--builtin_ntp_servers=127.21.42.212:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:56:54.666047 22561 tablet.cc:1721] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:54.667943 22470 raft_consensus.cc:491] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.668376 22470 raft_consensus.cc:513] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.670328 22470 leader_election.cc:290] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:56:54.672781 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3649588cf8414d9591e8f2ec3fb78e44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.674252 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:54.680009 22149 leader_election.cc:343] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.694633 22461 consensus_queue.cc:260] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.710124 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:54.712839 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_acf14fc56cfd4e89994ee97bd77f054e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.716755 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.720433 22559 mvcc.cc:204] Tried to move back new op lower bound from 7113409390342201344 to 7113409387438170112. Current Snapshot: MvccSnapshot[applied={T|T < 7113409390342201344}]
I20250112 08:56:54.724578 22565 tablet_replica.cc:331] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:54.725531 22565 raft_consensus.cc:2238] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:54.726042 22565 raft_consensus.cc:2267] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:54.727596 22565 ts_tablet_manager.cc:1905] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:56:54.731469 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acf14fc56cfd4e89994ee97bd77f054e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.736325 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]). Expected version 2 got 0
I20250112 08:56:54.733894 22560 tablet.cc:1721] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:54.742987 22146 leader_election.cc:304] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:54.743937 22535 raft_consensus.cc:2798] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:54.744287 22535 raft_consensus.cc:491] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:54.744601 22535 raft_consensus.cc:3054] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.744331 22571 tablet.cc:1721] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:54.749491 22456 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:54.749907 22456 ts_tablet_manager.cc:1397] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.242s user 0.014s sys 0.001s
I20250112 08:56:54.755861 22470 raft_consensus.cc:491] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.756350 22470 raft_consensus.cc:513] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.757047 22456 raft_consensus.cc:357] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.757675 22456 raft_consensus.cc:383] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:54.757931 22456 raft_consensus.cc:738] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:54.758385 22470 leader_election.cc:290] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.758424 22456 consensus_queue.cc:260] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.767450 22456 ts_tablet_manager.cc:1428] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20250112 08:56:54.770287 22456 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:54.770236 22535 raft_consensus.cc:513] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.772104 22535 leader_election.cc:290] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.772776 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:56:54.774067 22149 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.777736 22561 tablet.cc:1716] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:54.786310 21709 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef] alter complete (version 2)
I20250112 08:56:54.795393 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_67f935829cb54073963ebc60a9414e23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.797574 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67f935829cb54073963ebc60a9414e23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.801759 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_002caac6d09040a7bd7484ce1459d559" } new_table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" modify_external_catalogs: true
I20250112 08:56:54.804553 22456 tablet_bootstrap.cc:654] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:54.810879 22470 consensus_queue.cc:1035] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:54.812274 22470 consensus_queue.cc:1035] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:54.812494 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.805244 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_006c78dcc47545e1b7eb793c44633534 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 3 got 0
W20250112 08:56:54.823369 21999 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.000s sys 0.017s
I20250112 08:56:54.823257 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:54.823838 21999 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.865s user 0.000s sys 0.017s
W20250112 08:56:54.824433 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 866356us
W20250112 08:56:54.825387 22149 leader_election.cc:343] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.829335 21713 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b959aeabbb59496c93feb6f5fcc5b18c" } modify_external_catalogs: true
I20250112 08:56:54.829972 21713 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b959aeabbb59496c93feb6f5fcc5b18c" } modify_external_catalogs: true
I20250112 08:56:54.830415 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:54.831059 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 3 got 0
I20250112 08:56:54.831938 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:54.831845 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 (DEFAULT_TABLE table=default.Table_c3dc458b0d44410898f592553dccc2f4 [id=adee3342aa1141818220e48211f1bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.833086 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a679595edde4a1f9b7f0916b2d760f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.840471 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 69651ab160db41b3a187542ac1cd2839 (DEFAULT_TABLE table=default.Table_002caac6d09040a7bd7484ce1459d559 [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.841589 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69651ab160db41b3a187542ac1cd2839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.845754 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 (DEFAULT_TABLE table=default.Table_c3dc458b0d44410898f592553dccc2f4 [id=adee3342aa1141818220e48211f1bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.846813 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a679595edde4a1f9b7f0916b2d760f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.848356 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_4165821e9c5947b3bc177a3c85d8ff79" } modify_external_catalogs: true
I20250112 08:56:54.849004 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_4165821e9c5947b3bc177a3c85d8ff79" } modify_external_catalogs: true
I20250112 08:56:54.826964 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:54.854454 22067 raft_consensus.cc:2463] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:56:54.855355 22065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85059ac5ccd844e3893d214ef9ce6b9d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:54.855919 22065 raft_consensus.cc:3054] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.856002 22146 leader_election.cc:304] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:54.857100 22535 raft_consensus.cc:2798] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:54.857491 22535 raft_consensus.cc:491] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:54.857816 22535 raft_consensus.cc:3054] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.864001 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:54.866918 22535 raft_consensus.cc:513] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.867619 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48adce00f3644192880a8b16b5e514d9" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:54.868198 22213 raft_consensus.cc:3054] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:56:54.870200 22146 leader_election.cc:343] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.870594 22146 leader_election.cc:304] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:54.871460 22470 raft_consensus.cc:2743] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:54.876096 22535 leader_election.cc:290] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.878068 22213 raft_consensus.cc:2463] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
W20250112 08:56:54.888917 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69651ab160db41b3a187542ac1cd2839 consensus state has no leader
I20250112 08:56:54.892210 22496 consensus_queue.cc:1035] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:56:54.893024 21713 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 85059ac5ccd844e3893d214ef9ce6b9d
I20250112 08:56:54.894744 21999 leader_election.cc:304] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:54.898583 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:54.899693 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:54.903475 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 69651ab160db41b3a187542ac1cd2839 (DEFAULT_TABLE table=default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.853576 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 (DEFAULT_TABLE table=default.Table_c3dc458b0d44410898f592553dccc2f4 [id=adee3342aa1141818220e48211f1bf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.904870 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69651ab160db41b3a187542ac1cd2839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.910696 22042 tablet_service.cc:1467] Processing CreateTablet for tablet 69651ab160db41b3a187542ac1cd2839 (DEFAULT_TABLE table=default.Table_002caac6d09040a7bd7484ce1459d559 [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.910828 22496 raft_consensus.cc:2798] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:54.920643 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69651ab160db41b3a187542ac1cd2839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.921216 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_de19595082ed4dc890bbceb724ea6b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:54.923314 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de19595082ed4dc890bbceb724ea6b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.920686 22496 raft_consensus.cc:695] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:54.928990 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_acf14fc56cfd4e89994ee97bd77f054e" } new_table_name: "default.Table_2789759d2c4b4a5ea6ed786b5bc226f0" modify_external_catalogs: true
I20250112 08:56:54.929459 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a679595edde4a1f9b7f0916b2d760f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.929294 22496 consensus_queue.cc:237] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:54.931720 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:54.932353 22066 raft_consensus.cc:3054] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:54.939270 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:54.944142 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.945408 22337 tablet_service.cc:1467] Processing CreateTablet for tablet ef66f5e785864d1c845f5c5a5f944473 (DEFAULT_TABLE table=default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.946529 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef66f5e785864d1c845f5c5a5f944473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.950040 22189 tablet_service.cc:1467] Processing CreateTablet for tablet ef66f5e785864d1c845f5c5a5f944473 (DEFAULT_TABLE table=default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:54.951413 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef66f5e785864d1c845f5c5a5f944473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.963261 22561 tablet.cc:1721] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:54.966048 22535 raft_consensus.cc:491] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:54.966369 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:54.966472 22535 raft_consensus.cc:513] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:54.968075 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:54.968369 22535 leader_election.cc:290] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:54.973043 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:54.973941 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:54.975587 22338 tablet_service.cc:1467] Processing CreateTablet for tablet c08341b5ce6040d5be30f324d0373e38 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:54.978128 22149 leader_election.cc:343] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:54.984390 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08341b5ce6040d5be30f324d0373e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.987151 22580 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 08:56:54.988634 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 85059ac5ccd844e3893d214ef9ce6b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85059ac5ccd844e3893d214ef9ce6b9d already in progress: creating tablet
W20250112 08:56:54.989383 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 745d1a12356244fa811d4b0434ad1295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 745d1a12356244fa811d4b0434ad1295 already in progress: creating tablet
I20250112 08:56:54.999550 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 7451e83f3b554936a79c6724ebe76cbb (DEFAULT_TABLE table=default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.000777 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7451e83f3b554936a79c6724ebe76cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:54.995638 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]). Expected version 1 got 0
I20250112 08:56:54.960248 22065 raft_consensus.cc:2463] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
W20250112 08:56:55.023438 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93be4e0fc31c429f860c1a0cec64849b has no consensus state
I20250112 08:56:54.989459 22519 raft_consensus.cc:491] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:55.027027 22519 raft_consensus.cc:513] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.029227 22519 leader_election.cc:290] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:56:54.990525 22149 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.030871 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_006c78dcc47545e1b7eb793c44633534 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 3 got 0
I20250112 08:56:55.032166 22582 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:55.021541 22188 tablet_service.cc:1467] Processing CreateTablet for tablet c08341b5ce6040d5be30f324d0373e38 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.022747 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 7451e83f3b554936a79c6724ebe76cbb (DEFAULT_TABLE table=default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.035072 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fccd34473a6438389cda0b77391c8d7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:55.025772 22456 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:55.036334 22456 ts_tablet_manager.cc:1397] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.266s user 0.020s sys 0.021s
W20250112 08:56:55.036401 22149 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.040148 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08341b5ce6040d5be30f324d0373e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.041785 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7451e83f3b554936a79c6724ebe76cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.062350 22193 tablet_service.cc:1467] Processing CreateTablet for tablet a090d258647b4dd7bd91cf64011735dd (DEFAULT_TABLE table=default.Table_bb669f28ccf14e97b75d2d4253cc9ed4 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.065961 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a090d258647b4dd7bd91cf64011735dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.038798 22456 raft_consensus.cc:357] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.073995 22456 raft_consensus.cc:383] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.074339 22456 raft_consensus.cc:738] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.074393 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:56:55.075062 22456 consensus_queue.cc:260] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.077111 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:55.078760 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:56:55.079025 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet f252f0ec641d4bf5b4fe1399a8ced51e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f252f0ec641d4bf5b4fe1399a8ced51e already in progress: creating tablet
I20250112 08:56:55.087489 22456 ts_tablet_manager.cc:1428] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.051s user 0.004s sys 0.003s
I20250112 08:56:55.071781 22557 ts_tablet_manager.cc:1918] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:55.087922 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 003f1f1414584353a86214d27401291e (DEFAULT_TABLE table=default.Table_75e25b72370a4eeaa436c12df91c5fd0 [id=5dc9363aa79749eb9f04e9876f12a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.089540 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003f1f1414584353a86214d27401291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.095669 22456 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:55.101297 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_05ba113b55b64688b747d0552905aab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:55.103770 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05ba113b55b64688b747d0552905aab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:56:55.108628 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 88148.9us
I20250112 08:56:55.110977 21713 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_67f935829cb54073963ebc60a9414e23" } modify_external_catalogs: true
I20250112 08:56:55.113157 22557 log.cc:1198] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:56:55.113955 22557 ts_tablet_manager.cc:1939] T b0192d66e64d4d39b4bcfd6565dff723 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:55.113811 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bb669f28ccf14e97b75d2d4253cc9ed4" } new_table_name: "default.Table_cf768e89c13043eb9fcc7d154cd14518" modify_external_catalogs: true
I20250112 08:56:55.115298 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 003f1f1414584353a86214d27401291e (DEFAULT_TABLE table=default.Table_75e25b72370a4eeaa436c12df91c5fd0 [id=5dc9363aa79749eb9f04e9876f12a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.116418 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003f1f1414584353a86214d27401291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.118007 21713 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_67f935829cb54073963ebc60a9414e23" } modify_external_catalogs: true
I20250112 08:56:55.120044 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_006c78dcc47545e1b7eb793c44633534 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 3 got 0
I20250112 08:56:55.119232 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:55.121722 22047 tablet_service.cc:1467] Processing CreateTablet for tablet ef66f5e785864d1c845f5c5a5f944473 (DEFAULT_TABLE table=default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.130642 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 7451e83f3b554936a79c6724ebe76cbb (DEFAULT_TABLE table=default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.132503 22048 tablet_service.cc:1467] Processing CreateTablet for tablet a090d258647b4dd7bd91cf64011735dd (DEFAULT_TABLE table=default.Table_bb669f28ccf14e97b75d2d4253cc9ed4 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.142076 21709 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448] alter complete (version 1)
I20250112 08:56:55.142333 22569 tablet.cc:1721] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:55.111276 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fccd34473a6438389cda0b77391c8d7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:55.137035 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef66f5e785864d1c845f5c5a5f944473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.159591 22458 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:55.159338 22041 tablet_service.cc:1467] Processing CreateTablet for tablet c08341b5ce6040d5be30f324d0373e38 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.161350 22146 leader_election.cc:304] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:55.164492 22039 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:55.164484 22458 ts_tablet_manager.cc:1397] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 1.088s user 0.019s sys 0.017s
W20250112 08:56:55.167891 22146 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.168754 22146 leader_election.cc:304] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:55.170444 22146 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.170897 22146 leader_election.cc:304] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:55.171856 22519 raft_consensus.cc:2798] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:55.172559 22535 raft_consensus.cc:2743] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:55.173123 22535 raft_consensus.cc:2743] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:55.174226 22519 raft_consensus.cc:695] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:55.175072 22519 consensus_queue.cc:237] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.177237 22557 tablet_replica.cc:331] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:55.178093 22557 raft_consensus.cc:2238] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.178555 22557 raft_consensus.cc:2267] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.181537 22557 ts_tablet_manager.cc:1905] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.169196 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet b0192d66e64d4d39b4bcfd6565dff723 (table default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]) successfully deleted
I20250112 08:56:55.171257 22456 tablet_bootstrap.cc:654] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:55.187980 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 6c683c8c64a441e8b5be52b077fdcb08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c683c8c64a441e8b5be52b077fdcb08 already in progress: creating tablet
I20250112 08:56:55.192643 21713 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c1b1512c987e45f6b1fe361fb75996e3
I20250112 08:56:55.199272 22560 tablet.cc:1716] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:55.208292 22334 tablet_service.cc:1467] Processing CreateTablet for tablet a090d258647b4dd7bd91cf64011735dd (DEFAULT_TABLE table=default.Table_bb669f28ccf14e97b75d2d4253cc9ed4 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.210388 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a090d258647b4dd7bd91cf64011735dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:55.210987 22045 consensus_meta.cc:287] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 1.074s user 0.013s sys 0.004s
I20250112 08:56:55.212059 22045 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56484 (request call id 29) took 1149 ms. Trace:
I20250112 08:56:55.212445 22045 rpcz_store.cc:276] 0112 08:56:54.062424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:54.063004 (+ 580us) service_pool.cc:225] Handling call
0112 08:56:54.066014 (+ 3010us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 08:56:54.066108 (+ 94us) ts_tablet_manager.cc:676] Creating new metadata...
0112 08:56:54.091166 (+ 25058us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.211464 (+1120298us) tablet_replica.cc:206] Creating consensus instance
0112 08:56:55.211989 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 08:56:55.152491 22066 raft_consensus.cc:2463] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:56:55.216347 22461 ts_tablet_manager.cc:1428] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.568s user 0.005s sys 0.001s
W20250112 08:56:55.216650 22461 ts_tablet_manager.cc:1460] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Tablet startup took 568ms
I20250112 08:56:55.180105 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 003f1f1414584353a86214d27401291e (DEFAULT_TABLE table=default.Table_75e25b72370a4eeaa436c12df91c5fd0 [id=5dc9363aa79749eb9f04e9876f12a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.210194 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a090d258647b4dd7bd91cf64011735dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.093904 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_ca0708b2b934445fbd13bbf81e61f61f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:55.229315 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_c3dc458b0d44410898f592553dccc2f4" } new_table_name: "default.Table_79482f2a207c4ad2ae7463df893ec61e" modify_external_catalogs: true
W20250112 08:56:55.231489 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca0708b2b934445fbd13bbf81e61f61f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.233047 22146 leader_election.cc:304] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:55.234226 22519 raft_consensus.cc:2798] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:55.235658 22519 raft_consensus.cc:695] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:56:55.236495 22519 consensus_queue.cc:237] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.242141 22458 raft_consensus.cc:357] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.242811 22458 raft_consensus.cc:383] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.243073 22458 raft_consensus.cc:738] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.243016 22557 ts_tablet_manager.cc:1918] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 08:56:55.249244 22043 consensus_meta.cc:287] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 0.676s user 0.007s sys 0.000s
I20250112 08:56:55.243618 22458 consensus_queue.cc:260] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.257503 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08341b5ce6040d5be30f324d0373e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.270890 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003f1f1414584353a86214d27401291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.273243 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_9420682ee5cb40e893640510c8fafecb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:55.274961 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9420682ee5cb40e893640510c8fafecb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.284638 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7451e83f3b554936a79c6724ebe76cbb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:55.298492 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a090d258647b4dd7bd91cf64011735dd consensus state has no leader
I20250112 08:56:55.299782 21717 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_fb4954b383d44b93a4cbf55356bda5e4" } modify_external_catalogs: true
I20250112 08:56:55.300325 21717 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_fb4954b383d44b93a4cbf55356bda5e4" } modify_external_catalogs: true
I20250112 08:56:55.304631 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_5ca837d5572348a9841f503e80c7f126"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:55.312955 22458 ts_tablet_manager.cc:1428] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.148s user 0.006s sys 0.000s
W20250112 08:56:55.313783 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ca837d5572348a9841f503e80c7f126 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.329530 22565 ts_tablet_manager.cc:1918] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:55.329926 22565 log.cc:1198] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/f152b58bd9114691bab99c23b51a5e89
W20250112 08:56:55.329999 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a679595edde4a1f9b7f0916b2d760f2 consensus state has no leader
I20250112 08:56:55.330762 22565 ts_tablet_manager.cc:1939] T f152b58bd9114691bab99c23b51a5e89 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:55.336851 22458 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:55.340876 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]) successfully deleted
I20250112 08:56:55.336191 22557 log.cc:1198] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:55.349352 22557 ts_tablet_manager.cc:1939] T 6fccd34473a6438389cda0b77391c8d7 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:55.354799 22557 tablet_replica.cc:331] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:55.355671 22557 raft_consensus.cc:2238] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.356132 22557 raft_consensus.cc:2267] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.358623 22557 ts_tablet_manager.cc:1905] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.368659 22577 mvcc.cc:204] Tried to move back new op lower bound from 7113409388680933376 to 7113409386717560832. Current Snapshot: MvccSnapshot[applied={T|T < 7113409388680933376}]
I20250112 08:56:55.380914 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 6fccd34473a6438389cda0b77391c8d7 (table default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]) successfully deleted
W20250112 08:56:55.216908 22461 ts_tablet_manager.cc:1462] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Trace:
0112 08:56:52.519349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:52.519477 (+ 128us) service_pool.cc:225] Handling call
0112 08:56:52.534411 (+ 14934us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 08:56:52.534525 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0112 08:56:52.689397 (+154872us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:52.789709 (+100312us) tablet_replica.cc:206] Creating consensus instance
0112 08:56:52.790120 (+ 411us) inbound_call.cc:173] Queueing success response
0112 08:56:54.220340 (+1430220us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0112 08:56:54.648506 (+428166us) ts_tablet_manager.cc:1429] Starting tablet replica
0112 08:56:54.649064 (+ 558us) tablet_replica.cc:260] Starting instrumentation
0112 08:56:54.649479 (+ 415us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":247831,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"tablet-open.queue_time_us":1430278}
I20250112 08:56:55.389125 22461 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:55.409482 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet e43f8e3061c54472a80f3dad3d2b8d86
I20250112 08:56:55.411520 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:55.411943 22565 tablet_replica.cc:331] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:55.413089 22556 tablet_replica.cc:331] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:55.413498 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:55.413862 22556 raft_consensus.cc:2238] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.414445 22556 raft_consensus.cc:2267] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.414295 21717 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8e031f733adc400f80ef5a314e2c2704" } modify_external_catalogs: true
W20250112 08:56:55.414634 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet b0192d66e64d4d39b4bcfd6565dff723 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0192d66e64d4d39b4bcfd6565dff723 already in progress: creating tablet
I20250112 08:56:55.414817 21717 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8e031f733adc400f80ef5a314e2c2704" } modify_external_catalogs: true
W20250112 08:56:55.415254 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 6fccd34473a6438389cda0b77391c8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fccd34473a6438389cda0b77391c8d7 already in progress: creating tablet
I20250112 08:56:55.413499 22565 raft_consensus.cc:2238] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.416939 22565 raft_consensus.cc:2267] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.417337 22556 ts_tablet_manager.cc:1905] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.423210 22590 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:55.423885 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:55.428056 22458 tablet_bootstrap.cc:654] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:55.433251 22565 ts_tablet_manager.cc:1905] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.434616 22580 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37418 (request call id 31) took 1417 ms. Trace:
W20250112 08:56:55.415819 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 6c683c8c64a441e8b5be52b077fdcb08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c683c8c64a441e8b5be52b077fdcb08 already in progress: creating tablet
I20250112 08:56:55.444886 22561 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:55.454097 22456 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:55.454550 22456 ts_tablet_manager.cc:1397] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.360s user 0.009s sys 0.006s
I20250112 08:56:55.459122 22456 raft_consensus.cc:357] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.459829 22456 raft_consensus.cc:383] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.460109 22456 raft_consensus.cc:738] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.460757 22456 consensus_queue.cc:260] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.466408 22561 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37418 (request call id 34) took 1293 ms. Trace:
I20250112 08:56:55.492084 22569 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56484 (request call id 28) took 1512 ms. Trace:
I20250112 08:56:55.470468 22561 rpcz_store.cc:276] 0112 08:56:54.172638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:54.172781 (+ 143us) service_pool.cc:225] Handling call
0112 08:56:55.466352 (+1293571us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:54.173795 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:56:54.174127 (+ 332us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:56:55.429865 (+1255738us) alter_schema_op.cc:66] Acquired schema lock
0112 08:56:55.429951 (+ 86us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:56:55.430065 (+ 114us) alter_schema_op.cc:128] START. Timestamp: 1736672215408359
0112 08:56:55.430085 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 08:56:55.430568 (+ 483us) log.cc:844] Serialized 173 byte log entry
0112 08:56:55.444257 (+ 13689us) op_driver.cc:464] REPLICATION: finished
0112 08:56:55.444810 (+ 553us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:56:55.445985 (+ 1175us) log.cc:844] Serialized 5 byte log entry
0112 08:56:55.457930 (+ 11945us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.464218 (+ 6288us) log.cc:844] Serialized 14 byte log entry
0112 08:56:55.466232 (+ 2014us) alter_schema_op.cc:73] Released schema lock
0112 08:56:55.466261 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":396,"prepare.queue_time_us":150,"prepare.run_cpu_time_us":87595,"prepare.run_wall_time_us":1258728,"raft.queue_time_us":80,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":637,"replication_time_us":14069,"spinlock_wait_cycles":1280}]]}
I20250112 08:56:55.506590 22561 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:56:55.435055 22580 rpcz_store.cc:276] 0112 08:56:54.017699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:54.017897 (+ 198us) service_pool.cc:225] Handling call
0112 08:56:55.434557 (+1416660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:54.040851 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:56:54.041314 (+ 463us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:56:54.041352 (+ 38us) alter_schema_op.cc:66] Acquired schema lock
0112 08:56:54.041470 (+ 118us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:56:54.044949 (+ 3479us) alter_schema_op.cc:128] START. Timestamp: 1736672214033431
0112 08:56:54.044978 (+ 29us) op_driver.cc:348] REPLICATION: starting
0112 08:56:54.045961 (+ 983us) log.cc:844] Serialized 173 byte log entry
0112 08:56:54.980511 (+934550us) op_driver.cc:464] REPLICATION: finished
0112 08:56:54.986361 (+ 5850us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:56:55.058978 (+ 72617us) log.cc:844] Serialized 5 byte log entry
0112 08:56:55.416671 (+357693us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.429141 (+ 12470us) log.cc:844] Serialized 14 byte log entry
0112 08:56:55.429797 (+ 656us) alter_schema_op.cc:73] Released schema lock
0112 08:56:55.429830 (+ 33us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5672,"prepare.queue_time_us":17877,"prepare.run_cpu_time_us":1964,"prepare.run_wall_time_us":10566,"raft.queue_time_us":169,"raft.run_cpu_time_us":1099,"raft.run_wall_time_us":1099,"replication_time_us":935429,"thread_start_us":30422,"threads_started":2}]]}
I20250112 08:56:55.517374 22458 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:55.517769 22458 ts_tablet_manager.cc:1397] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.181s user 0.006s sys 0.007s
I20250112 08:56:55.517802 22582 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:55.492436 22569 rpcz_store.cc:276] 0112 08:56:53.980133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:53.993680 (+ 13547us) service_pool.cc:225] Handling call
0112 08:56:55.491981 (+1498301us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:54.022832 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:56:54.023205 (+ 373us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:56:54.023235 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0112 08:56:54.023314 (+ 79us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:56:54.023452 (+ 138us) alter_schema_op.cc:128] START. Timestamp: 1736672214023364
0112 08:56:54.023476 (+ 24us) op_driver.cc:348] REPLICATION: starting
0112 08:56:54.182732 (+159256us) log.cc:844] Serialized 173 byte log entry
0112 08:56:54.185046 (+ 2314us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54000b09d4. stack: 00007f8983942853 00007f8983942750 00007f89834b7584 00000000004fcd24 00000000004fccb9 00007f898a8426d1 00007f898a8b9198 00007f898ad253b1 00007f898ad24b88 00007f898ad29821 00007f898ad297d9 00007f898ad29769 00007f898ad29731 00007f898ad2895d 000000000059ea04 000000000059e839
0112 08:56:54.574518 (+389472us) op_driver.cc:464] REPLICATION: finished
0112 08:56:55.115778 (+541260us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:56:55.261375 (+145597us) log.cc:844] Serialized 5 byte log entry
0112 08:56:55.483403 (+222028us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.484582 (+ 1179us) log.cc:844] Serialized 14 byte log entry
0112 08:56:55.491235 (+ 6653us) alter_schema_op.cc:73] Released schema lock
0112 08:56:55.491272 (+ 37us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":443009,"prepare.queue_time_us":27384,"prepare.run_cpu_time_us":6224,"prepare.run_wall_time_us":162437,"raft.queue_time_us":261,"raft.run_cpu_time_us":555,"raft.run_wall_time_us":555,"replication_time_us":550841,"spinlock_wait_cycles":364526208,"thread_start_us":260553,"threads_started":3,"wal-append.queue_time_us":93129}]]}
I20250112 08:56:55.520059 22557 ts_tablet_manager.cc:1918] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:55.520444 22557 log.cc:1198] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:55.521016 22569 tablet.cc:1721] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:55.524902 22458 raft_consensus.cc:357] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.524389 22592 tablet.cc:1721] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:55.525600 22458 raft_consensus.cc:383] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.525879 22458 raft_consensus.cc:738] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.526451 22458 consensus_queue.cc:260] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.532075 22458 ts_tablet_manager.cc:1428] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250112 08:56:55.545341 22461 tablet_bootstrap.cc:654] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:55.545221 22590 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:55.547361 22565 ts_tablet_manager.cc:1918] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:55.547755 22565 log.cc:1198] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/85059ac5ccd844e3893d214ef9ce6b9d
I20250112 08:56:55.548451 22565 ts_tablet_manager.cc:1939] T 85059ac5ccd844e3893d214ef9ce6b9d P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:55.556115 22582 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:56:55.564013 22596 raft_consensus.cc:491] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:55.564515 22596 raft_consensus.cc:513] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.568794 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1b6fafd4ef8b49beba9040f711b5dac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:55.570489 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b6fafd4ef8b49beba9040f711b5dac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.574323 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e" } new_table_name: "default.Table_20619b3c16604ae798fb5df3d56d5647" modify_external_catalogs: true
I20250112 08:56:55.576967 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_79482f2a207c4ad2ae7463df893ec61e" } new_table_name: "default.Table_3f79f09bbc3949a68c55ec5dab776291" modify_external_catalogs: true
I20250112 08:56:55.577486 22456 ts_tablet_manager.cc:1428] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.123s user 0.007s sys 0.000s
I20250112 08:56:55.578438 22456 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:55.578703 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 5157642d667d49aaad41df56be38801f (table default.Table_006c78dcc47545e1b7eb793c44633534 [id=e812d5ec0441481bb47d2aa1eeee407b]). Expected version 3 got 1
W20250112 08:56:55.587158 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_79482f2a207c4ad2ae7463df893ec61e [id=adee3342aa1141818220e48211f1bf25], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a679595edde4a1f9b7f0916b2d760f2 consensus state has no leader
I20250112 08:56:55.589293 22458 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:55.591137 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 745d1a12356244fa811d4b0434ad1295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:52 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:55.521122 22557 ts_tablet_manager.cc:1939] T 69a9c2b5a5d044c789a3eda07e8a312d P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:55.596836 22556 ts_tablet_manager.cc:1918] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:56:55.598824 22565 tablet_replica.cc:331] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:55.600777 22565 raft_consensus.cc:2238] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.601328 22565 raft_consensus.cc:2267] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.610190 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 93be4e0fc31c429f860c1a0cec64849b (DEFAULT_TABLE table=default.Table_2789759d2c4b4a5ea6ed786b5bc226f0 [id=2c86b027468242f694274397ec685267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.611229 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93be4e0fc31c429f860c1a0cec64849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.612694 22556 log.cc:1198] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/e43f8e3061c54472a80f3dad3d2b8d86
I20250112 08:56:55.612924 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 93be4e0fc31c429f860c1a0cec64849b (DEFAULT_TABLE table=default.Table_2789759d2c4b4a5ea6ed786b5bc226f0 [id=2c86b027468242f694274397ec685267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.596978 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet 1b75895a7c9040c1a2ce8c25aeb5c32b from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:55.613973 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93be4e0fc31c429f860c1a0cec64849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.617388 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 14c7056aaf8844da915acec5f9f92eeb (DEFAULT_TABLE table=default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.618526 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c7056aaf8844da915acec5f9f92eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.620007 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 6182470e64b14e329b278680dc132fcb (DEFAULT_TABLE table=default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e [id=847a5348757c4e10877a3ea2416cc60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.621587 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6182470e64b14e329b278680dc132fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.622382 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (DEFAULT_TABLE table=default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.623404 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b5b6ea6f044fc8dcaafdc2e587f84. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:55.624217 22290 leader_election.cc:343] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 745d1a12356244fa811d4b0434ad1295
I20250112 08:56:55.613381 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 85059ac5ccd844e3893d214ef9ce6b9d (table default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]) successfully deleted
I20250112 08:56:55.628540 22496 raft_consensus.cc:491] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:56:55.628793 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet f252f0ec641d4bf5b4fe1399a8ced51e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f252f0ec641d4bf5b4fe1399a8ced51e already in progress: creating tablet
I20250112 08:56:55.629015 22496 raft_consensus.cc:513] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.630791 22496 leader_election.cc:290] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:55.613512 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_94180f60c7fb492eb9658711bc6c492d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:55.614318 21712 master_service.cc:946] ReplaceTablet: received request to replace tablet 48adce00f3644192880a8b16b5e514d9 from {username='slave'} at 127.0.0.1:56650
W20250112 08:56:55.632773 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94180f60c7fb492eb9658711bc6c492d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.633459 22458 tablet_bootstrap.cc:654] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:55.634199 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:55.634788 22213 raft_consensus.cc:2463] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:55.638980 22591 raft_consensus.cc:491] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:55.639293 22591 raft_consensus.cc:513] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.639552 22338 tablet_service.cc:1467] Processing CreateTablet for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (DEFAULT_TABLE table=default.Table_39462b870602435cb6f626b7cbf102f6 [id=78a10e1e86624d089f2b6866a4a2da03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.640609 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72d2062bf2a4c79b5a8be974d1b8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.642038 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f252f0ec641d4bf5b4fe1399a8ced51e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:55.642762 21715 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]). Expected version 1 got 0
W20250112 08:56:55.643472 22149 leader_election.cc:343] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.607110 21708 catalog_manager.cc:5526] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:55.646202 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:55.647255 22591 leader_election.cc:290] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:55.647514 22596 leader_election.cc:290] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:55.649686 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 14c7056aaf8844da915acec5f9f92eeb (DEFAULT_TABLE table=default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.651326 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c7056aaf8844da915acec5f9f92eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.653576 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 6182470e64b14e329b278680dc132fcb (DEFAULT_TABLE table=default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e [id=847a5348757c4e10877a3ea2416cc60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.654547 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6182470e64b14e329b278680dc132fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.654973 21715 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3649588cf8414d9591e8f2ec3fb78e44" } modify_external_catalogs: true
I20250112 08:56:55.655632 21715 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3649588cf8414d9591e8f2ec3fb78e44" } modify_external_catalogs: true
I20250112 08:56:55.656584 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (DEFAULT_TABLE table=default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.657829 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b5b6ea6f044fc8dcaafdc2e587f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.611790 22461 tablet_bootstrap.cc:492] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:55.660223 22461 ts_tablet_manager.cc:1397] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.271s user 0.006s sys 0.004s
I20250112 08:56:55.662688 22461 raft_consensus.cc:357] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.608851 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 93be4e0fc31c429f860c1a0cec64849b (DEFAULT_TABLE table=default.Table_2789759d2c4b4a5ea6ed786b5bc226f0 [id=2c86b027468242f694274397ec685267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.664155 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93be4e0fc31c429f860c1a0cec64849b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.668586 22556 ts_tablet_manager.cc:1939] T e43f8e3061c54472a80f3dad3d2b8d86 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:55.671449 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]) successfully deleted
I20250112 08:56:55.672865 22556 tablet_replica.cc:331] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:55.674006 21999 leader_election.cc:304] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:55.677482 22496 raft_consensus.cc:2798] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:55.677860 22496 raft_consensus.cc:491] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:55.678373 22496 raft_consensus.cc:3054] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:55.634872 22569 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:56:55.681262 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f252f0ec641d4bf5b4fe1399a8ced51e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:55.682590 22146 leader_election.cc:343] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.683030 22146 leader_election.cc:304] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:55.684182 22591 raft_consensus.cc:2743] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:55.685180 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 14c7056aaf8844da915acec5f9f92eeb (DEFAULT_TABLE table=default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.685478 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 6182470e64b14e329b278680dc132fcb (DEFAULT_TABLE table=default.Table_73b2d3054bb24a88b89cc9c0c02fdd4e [id=847a5348757c4e10877a3ea2416cc60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.686342 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c7056aaf8844da915acec5f9f92eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.686983 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (DEFAULT_TABLE table=default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.688477 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b5b6ea6f044fc8dcaafdc2e587f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.689991 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6182470e64b14e329b278680dc132fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.708828 22456 tablet_bootstrap.cc:654] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:55.727483 22047 tablet_service.cc:1467] Processing CreateTablet for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (DEFAULT_TABLE table=default.Table_39462b870602435cb6f626b7cbf102f6 [id=78a10e1e86624d089f2b6866a4a2da03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.728608 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72d2062bf2a4c79b5a8be974d1b8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.729434 22590 tablet.cc:1716] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:55.741722 22565 ts_tablet_manager.cc:1905] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.742237 22192 tablet_service.cc:1467] Processing CreateTablet for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (DEFAULT_TABLE table=default.Table_39462b870602435cb6f626b7cbf102f6 [id=78a10e1e86624d089f2b6866a4a2da03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.745011 22496 raft_consensus.cc:513] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:55.746104 21703 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.253:41411, user_credentials={real_user=slave}} blocked reactor thread for 114787us
I20250112 08:56:55.746511 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72d2062bf2a4c79b5a8be974d1b8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.755319 22591 raft_consensus.cc:491] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:55.755784 22591 raft_consensus.cc:513] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:55.757608 22591 leader_election.cc:290] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:55.760483 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c683c8c64a441e8b5be52b077fdcb08" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:55.764937 22146 leader_election.cc:343] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.769249 22458 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:55.769630 22458 ts_tablet_manager.cc:1397] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.007s
I20250112 08:56:55.773157 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:55.773761 22213 raft_consensus.cc:3054] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:55.771811 22458 raft_consensus.cc:357] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.777839 22458 raft_consensus.cc:383] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.778071 22458 raft_consensus.cc:738] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.778780 22458 consensus_queue.cc:260] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.781927 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 69a9c2b5a5d044c789a3eda07e8a312d (table default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]) successfully deleted
I20250112 08:56:55.782728 22557 tablet_replica.cc:331] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:55.783502 22557 raft_consensus.cc:2238] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:55.784193 22557 pending_rounds.cc:62] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Trying to abort 1 pending ops.
I20250112 08:56:55.784420 22557 pending_rounds.cc:69] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409393279705088 op_type: NO_OP noop_request { }
I20250112 08:56:55.784772 22557 raft_consensus.cc:2883] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:56:55.785101 22557 raft_consensus.cc:2267] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.784904 22213 raft_consensus.cc:2463] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:55.786185 21999 leader_election.cc:304] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:56:55.787410 22047 tablet_service.cc:1467] Processing CreateTablet for tablet dabc37f08d1948118b98ba25b689bea4 (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.788445 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabc37f08d1948118b98ba25b689bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.675297 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:55.688222 22290 leader_election.cc:343] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 745d1a12356244fa811d4b0434ad1295
I20250112 08:56:55.792455 22290 leader_election.cc:304] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:55.793273 22596 raft_consensus.cc:2743] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:55.795703 22002 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.758124 22461 raft_consensus.cc:383] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:55.771251 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b75895a7c9040c1a2ce8c25aeb5c32b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:56:55.793284 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 105083us
W20250112 08:56:55.797379 22002 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.797890 22461 raft_consensus.cc:738] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:55.798712 22461 consensus_queue.cc:260] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:55.800931 22600 raft_consensus.cc:2798] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:55.801388 22600 raft_consensus.cc:695] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:55.802104 22600 consensus_queue.cc:237] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:55.807294 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6182470e64b14e329b278680dc132fcb consensus state has no leader
W20250112 08:56:55.810330 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a679595edde4a1f9b7f0916b2d760f2 consensus state has no leader
I20250112 08:56:55.818352 22192 tablet_service.cc:1467] Processing CreateTablet for tablet dabc37f08d1948118b98ba25b689bea4 (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.819445 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabc37f08d1948118b98ba25b689bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.798964 22335 tablet_service.cc:1467] Processing CreateTablet for tablet dabc37f08d1948118b98ba25b689bea4 (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:55.830296 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabc37f08d1948118b98ba25b689bea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:55.799834 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c683c8c64a441e8b5be52b077fdcb08" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:55.832039 22149 consensus_peers.cc:487] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e43f8e3061c54472a80f3dad3d2b8d86. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:56:55.832887 22149 leader_election.cc:343] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:55.833271 22149 leader_election.cc:304] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:55.834795 22591 raft_consensus.cc:2743] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:55.860463 22458 ts_tablet_manager.cc:1428] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.091s user 0.005s sys 0.002s
I20250112 08:56:55.861390 22565 ts_tablet_manager.cc:1918] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:56:55.861742 22565 log.cc:1198] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/e43f8e3061c54472a80f3dad3d2b8d86
I20250112 08:56:55.861831 22458 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:55.862445 22565 ts_tablet_manager.cc:1939] T e43f8e3061c54472a80f3dad3d2b8d86 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:55.864950 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]) successfully deleted
I20250112 08:56:55.870863 22557 ts_tablet_manager.cc:1905] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.800096 22556 raft_consensus.cc:2238] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.872949 22556 raft_consensus.cc:2267] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.873965 22496 leader_election.cc:290] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:55.884073 22461 ts_tablet_manager.cc:1428] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.223s user 0.004s sys 0.002s
I20250112 08:56:55.885210 22461 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:55.903908 22556 ts_tablet_manager.cc:1905] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.909363 22557 ts_tablet_manager.cc:1918] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:55.909731 22557 log.cc:1198] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/85059ac5ccd844e3893d214ef9ce6b9d
I20250112 08:56:55.910419 22557 ts_tablet_manager.cc:1939] T 85059ac5ccd844e3893d214ef9ce6b9d P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:56:55.910322 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 124219us
I20250112 08:56:55.912678 22456 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:55.913136 22456 ts_tablet_manager.cc:1397] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.335s user 0.006s sys 0.004s
I20250112 08:56:55.917796 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_d3832ac137d5425db9a0f35eb56cfd68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:55.919410 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3832ac137d5425db9a0f35eb56cfd68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.921423 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_7111dd9cee36485ab448639684e10659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:55.923286 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7111dd9cee36485ab448639684e10659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:55.928555 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 48) took 1029 ms. Trace:
I20250112 08:56:55.928929 22557 rpcz_store.cc:276] 0112 08:56:54.899345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:54.899505 (+ 160us) service_pool.cc:225] Handling call
0112 08:56:55.782562 (+883057us) ts_tablet_manager.cc:1145] Deleting tablet 85059ac5ccd844e3893d214ef9ce6b9d
0112 08:56:55.782607 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:55.886357 (+103750us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.909144 (+ 22787us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:55.928486 (+ 19342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":882356}
I20250112 08:56:55.930226 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 85059ac5ccd844e3893d214ef9ce6b9d (table default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]) successfully deleted
I20250112 08:56:55.936311 22557 tablet_replica.cc:331] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:55.937103 22557 raft_consensus.cc:2238] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:55.937541 22557 raft_consensus.cc:2267] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:55.939275 22557 ts_tablet_manager.cc:1905] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:55.974223 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 7451e83f3b554936a79c6724ebe76cbb
I20250112 08:56:55.976356 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:55.977703 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:55.978441 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:55.979458 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 7451e83f3b554936a79c6724ebe76cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7451e83f3b554936a79c6724ebe76cbb already in progress: creating tablet
I20250112 08:56:55.915472 22456 raft_consensus.cc:357] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.015412 22496 raft_consensus.cc:491] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.015553 22456 raft_consensus.cc:383] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.015846 22496 raft_consensus.cc:513] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.017673 22557 ts_tablet_manager.cc:1918] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:56.018070 22557 log.cc:1198] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:56:56.018705 22557 ts_tablet_manager.cc:1939] T f252f0ec641d4bf5b4fe1399a8ced51e P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:56.019501 22496 leader_election.cc:290] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:56:56.021591 21999 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:56.022516 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet f252f0ec641d4bf5b4fe1399a8ced51e (table default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]) successfully deleted
W20250112 08:56:56.023119 22146 consensus_peers.cc:487] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e43f8e3061c54472a80f3dad3d2b8d86. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:56.026063 22557 tablet_replica.cc:331] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:56.026947 22557 raft_consensus.cc:2238] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:56.015844 22456 raft_consensus.cc:738] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.028016 22557 raft_consensus.cc:2267] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.028846 22591 consensus_queue.cc:1035] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:56.028716 22456 consensus_queue.cc:260] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.030814 22461 tablet_bootstrap.cc:654] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.041910 22557 ts_tablet_manager.cc:1905] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.046427 22456 ts_tablet_manager.cc:1428] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.133s user 0.006s sys 0.001s
I20250112 08:56:56.049000 22456 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.055584 22496 consensus_queue.cc:1035] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:56.063934 22458 tablet_bootstrap.cc:654] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.065646 22461 tablet_bootstrap.cc:492] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.066107 22461 ts_tablet_manager.cc:1397] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.181s user 0.001s sys 0.008s
I20250112 08:56:56.068210 22461 raft_consensus.cc:357] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.068854 22461 raft_consensus.cc:383] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.069115 22461 raft_consensus.cc:738] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.104151 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_20619b3c16604ae798fb5df3d56d5647" } new_table_name: "default.Table_89c2768382e2459894ea3add3ee2319c" modify_external_catalogs: true
I20250112 08:56:56.104386 22461 consensus_queue.cc:260] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.123535 22557 ts_tablet_manager.cc:1918] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:56:56.124524 22557 log.cc:1198] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/e43f8e3061c54472a80f3dad3d2b8d86
I20250112 08:56:56.131736 22557 ts_tablet_manager.cc:1939] T e43f8e3061c54472a80f3dad3d2b8d86 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:56.133591 22461 ts_tablet_manager.cc:1428] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.067s user 0.006s sys 0.001s
I20250112 08:56:56.134327 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet e43f8e3061c54472a80f3dad3d2b8d86 (table default.Table_fb4954b383d44b93a4cbf55356bda5e4 [id=964360620b2744a49ced4e4bf2ddd8ef]) successfully deleted
W20250112 08:56:56.138185 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 7451e83f3b554936a79c6724ebe76cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7451e83f3b554936a79c6724ebe76cbb already in progress: creating tablet
I20250112 08:56:56.142822 22456 tablet_bootstrap.cc:654] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.152825 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fccd34473a6438389cda0b77391c8d7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:56.153859 22357 raft_consensus.cc:3054] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:56:56.157461 22149 consensus_peers.cc:487] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870 -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:56:56.158252 22002 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.158633 22002 leader_election.cc:304] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:56.168083 22609 raft_consensus.cc:2743] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:56.171964 22596 raft_consensus.cc:491] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.172387 22596 raft_consensus.cc:513] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.174103 22596 leader_election.cc:290] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.177809 22609 consensus_queue.cc:1035] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:56:56.180732 22456 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.181146 22456 ts_tablet_manager.cc:1397] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.001s
I20250112 08:56:56.183709 22456 raft_consensus.cc:357] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.184381 22456 raft_consensus.cc:383] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.184623 22456 raft_consensus.cc:738] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:56:56.186285 22290 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 85059ac5ccd844e3893d214ef9ce6b9d
W20250112 08:56:56.188444 22290 leader_election.cc:343] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 85059ac5ccd844e3893d214ef9ce6b9d
I20250112 08:56:56.188822 22290 leader_election.cc:304] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:56.189811 22596 raft_consensus.cc:2743] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:56.185209 22456 consensus_queue.cc:260] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.198154 22456 ts_tablet_manager.cc:1428] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20250112 08:56:56.198987 22456 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.201450 22461 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:56.213563 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_2789759d2c4b4a5ea6ed786b5bc226f0" } new_table_name: "default.Table_8dd89a8dac4a4501a9cbe3108faab913" modify_external_catalogs: true
I20250112 08:56:56.217244 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 48adce00f3644192880a8b16b5e514d9
I20250112 08:56:56.218698 21712 catalog_manager.cc:6596] ReplaceTablet: tablet 48adce00f3644192880a8b16b5e514d9 deleted and replaced by tablet 1c1c634513e14e818601894834c41378
I20250112 08:56:56.219724 22461 tablet_bootstrap.cc:654] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.219717 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 48adce00f3644192880a8b16b5e514d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c1c634513e14e818601894834c41378) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:56.220383 22557 tablet_replica.cc:331] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:56.221086 22557 raft_consensus.cc:2238] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.221531 22557 raft_consensus.cc:2267] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.219727 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_96450631653649f3b688e38443936e3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:56.223412 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96450631653649f3b688e38443936e3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:56.223737 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 48adce00f3644192880a8b16b5e514d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c1c634513e14e818601894834c41378) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.224457 22565 tablet_replica.cc:331] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:56.225126 22557 ts_tablet_manager.cc:1905] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.225186 22565 raft_consensus.cc:2238] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:56.225955 22565 raft_consensus.cc:2267] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.227993 22565 ts_tablet_manager.cc:1905] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.230548 22470 raft_consensus.cc:491] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.230958 22470 raft_consensus.cc:513] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.233177 22470 leader_election.cc:290] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.233613 22456 tablet_bootstrap.cc:654] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.235273 22461 tablet_bootstrap.cc:492] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.235707 22461 ts_tablet_manager.cc:1397] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.006s
I20250112 08:56:56.236461 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:56.237731 22146 leader_election.cc:343] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.238065 22461 raft_consensus.cc:357] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.238711 22461 raft_consensus.cc:383] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.238991 22461 raft_consensus.cc:738] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.238715 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:56.240617 22149 leader_election.cc:343] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.241009 22149 leader_election.cc:304] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:56.242059 22470 raft_consensus.cc:2743] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:56.240839 22461 consensus_queue.cc:260] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.230080 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 48adce00f3644192880a8b16b5e514d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c1c634513e14e818601894834c41378) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:56.247876 22461 ts_tablet_manager.cc:1428] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.012s user 0.003s sys 0.001s
I20250112 08:56:56.255596 22557 ts_tablet_manager.cc:1918] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:56.256207 22565 ts_tablet_manager.cc:1918] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:56.256656 22461 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:56.256639 22565 log.cc:1198] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/48adce00f3644192880a8b16b5e514d9
I20250112 08:56:56.257404 22565 ts_tablet_manager.cc:1939] T 48adce00f3644192880a8b16b5e514d9 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:56.260691 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]) successfully deleted
I20250112 08:56:56.267891 22461 tablet_bootstrap.cc:654] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.277657 22557 log.cc:1198] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/48adce00f3644192880a8b16b5e514d9
I20250112 08:56:56.277812 22596 raft_consensus.cc:491] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.278458 22557 ts_tablet_manager.cc:1939] T 48adce00f3644192880a8b16b5e514d9 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:56.278260 22596 raft_consensus.cc:513] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.269721 22456 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.279291 22456 ts_tablet_manager.cc:1397] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.003s
I20250112 08:56:56.280035 22596 leader_election.cc:290] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.281883 22456 raft_consensus.cc:357] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.282524 22456 raft_consensus.cc:383] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.282799 22456 raft_consensus.cc:738] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.283340 22456 consensus_queue.cc:260] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.291994 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:56.293695 21709 catalog_manager.cc:6596] ReplaceTablet: tablet 1b75895a7c9040c1a2ce8c25aeb5c32b deleted and replaced by tablet 85030734daed45d5b67dbffeaa3b8086
I20250112 08:56:56.296682 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b with delete_type TABLET_DATA_DELETED (replaced by tablet 85030734daed45d5b67dbffeaa3b8086) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:56.297912 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b with delete_type TABLET_DATA_DELETED (replaced by tablet 85030734daed45d5b67dbffeaa3b8086) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.298583 22565 tablet_replica.cc:331] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:56.299319 22565 raft_consensus.cc:2238] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:56.299980 22565 pending_rounds.cc:62] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Trying to abort 1 pending ops.
I20250112 08:56:56.300201 22565 pending_rounds.cc:69] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409395935531008 op_type: NO_OP noop_request { }
I20250112 08:56:56.300616 22565 raft_consensus.cc:2883] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:56:56.300936 22565 raft_consensus.cc:2267] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.305734 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]) successfully deleted
I20250112 08:56:56.306464 22565 ts_tablet_manager.cc:1905] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.307004 22456 ts_tablet_manager.cc:1428] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250112 08:56:56.307653 22557 tablet_replica.cc:331] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:56.308303 22557 raft_consensus.cc:2238] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.308699 22557 raft_consensus.cc:2267] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.311339 22456 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.314468 22609 raft_consensus.cc:491] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.314916 22609 raft_consensus.cc:513] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.317754 22609 leader_election.cc:290] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:56.317658 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:56.319002 22002 leader_election.cc:343] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:56.319563 22290 leader_election.cc:343] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: f152b58bd9114691bab99c23b51a5e89
I20250112 08:56:56.320590 22557 ts_tablet_manager.cc:1905] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:56:56.322228 22290 leader_election.cc:343] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: f152b58bd9114691bab99c23b51a5e89
I20250112 08:56:56.322607 22290 leader_election.cc:304] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:56.323570 22596 raft_consensus.cc:2743] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:56.324417 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:56.325026 22213 raft_consensus.cc:2463] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:56.326179 21999 leader_election.cc:304] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:56.334045 22609 raft_consensus.cc:2798] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:56.334415 22609 raft_consensus.cc:491] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:56.334798 22609 raft_consensus.cc:3054] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:56.336179 22456 tablet_bootstrap.cc:654] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.340229 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b with delete_type TABLET_DATA_DELETED (replaced by tablet 85030734daed45d5b67dbffeaa3b8086) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:56.349078 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]). Expected version 1 got 0
I20250112 08:56:56.352239 21710 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_006c78dcc47545e1b7eb793c44633534 [id=e812d5ec0441481bb47d2aa1eeee407b] alter complete (version 3)
I20250112 08:56:56.349315 22565 ts_tablet_manager.cc:1918] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 08:56:56.352916 21710 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 24) took 757 ms (client timeout 1000 ms). Trace:
I20250112 08:56:56.353202 22565 log.cc:1198] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:56.354506 22565 ts_tablet_manager.cc:1939] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:56.358417 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (table default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]) successfully deleted
W20250112 08:56:56.353329 21710 rpcz_store.cc:269] 0112 08:56:55.595067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:55.595196 (+ 129us) service_pool.cc:225] Handling call
0112 08:56:55.601797 (+ 6601us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26572dee 00007fcf242d0fc4 00007fcf242d0f52 00007fcf242d0e81 00007fcf242d0dfc 00007fcf242d00b2 00007fcf2259adfc 00007fcf2259a236
0112 08:56:56.352853 (+751056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:55.826343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:55.826727 (+ 384us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:55.826752 (+ 25us) write_op.cc:435] Acquired schema lock
0112 08:56:55.826764 (+ 12us) tablet.cc:659] Decoding operations
0112 08:56:55.827023 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:55.827061 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:55.827082 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:56:55.827222 (+ 140us) tablet.cc:698] Row locks acquired
0112 08:56:55.827238 (+ 16us) write_op.cc:260] PREPARE: finished
0112 08:56:55.889785 (+ 62547us) write_op.cc:270] Start()
0112 08:56:55.889871 (+ 86us) write_op.cc:276] Timestamp: P: 1736672215889721 usec, L: 0
0112 08:56:55.889885 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:56:55.933344 (+ 43459us) log.cc:844] Serialized 355 byte log entry
0112 08:56:55.941801 (+ 8457us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf247526d1 00007fcf247c9198 00007fcf24c353b1 00007fcf24c34b88 00007fcf24c39821 00007fcf24c397d9 00007fcf24c39769 00007fcf24c39731 00007fcf24c3895d 000000000059ea04 000000000059e839
0112 08:56:55.952158 (+ 10357us) op_driver.cc:464] REPLICATION: finished
0112 08:56:55.954518 (+ 2360us) write_op.cc:301] APPLY: starting
0112 08:56:55.954572 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:55.954658 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:55.954687 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:55.954840 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:55.955004 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:56:55.955051 (+ 47us) write_op.cc:312] APPLY: finished
0112 08:56:55.955231 (+ 180us) log.cc:844] Serialized 22 byte log entry
0112 08:56:55.955869 (+ 638us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:55.955973 (+ 104us) write_op.cc:454] Released schema lock
0112 08:56:55.955999 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 08:56:55.956354 (+ 355us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:55.956384 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":12516736,"child_traces":[["op",{"apply.queue_time_us":2177,"apply.run_cpu_time_us":1239,"apply.run_wall_time_us":1830,"num_ops":1,"prepare.queue_time_us":220914,"prepare.run_cpu_time_us":2996,"prepare.run_wall_time_us":125401,"raft.queue_time_us":140,"raft.run_cpu_time_us":851,"raft.run_wall_time_us":851,"replication_time_us":62190,"spinlock_wait_cycles":98781056}]]}
I20250112 08:56:56.361287 21709 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_1eb512e21fe24840b674e404ed647af7" } modify_external_catalogs: true
I20250112 08:56:56.362195 21709 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_1eb512e21fe24840b674e404ed647af7" } modify_external_catalogs: true
I20250112 08:56:56.368681 22456 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.368911 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b with delete_type TABLET_DATA_DELETED (replaced by tablet 85030734daed45d5b67dbffeaa3b8086) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.369077 22456 ts_tablet_manager.cc:1397] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.003s
I20250112 08:56:56.369218 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet e43f8e3061c54472a80f3dad3d2b8d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.370618 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 48adce00f3644192880a8b16b5e514d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c1c634513e14e818601894834c41378) from {username='slave'} at 127.0.0.1:56484
W20250112 08:56:56.374235 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b because the tablet was not found. No further retry: Not found: Tablet not found: 1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:56.374689 22557 ts_tablet_manager.cc:1918] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:56.375028 22557 log.cc:1198] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:56.375589 22557 ts_tablet_manager.cc:1939] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:56:56.376160 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet e43f8e3061c54472a80f3dad3d2b8d86 because the tablet was not found. No further retry: Not found: Tablet not found: e43f8e3061c54472a80f3dad3d2b8d86
W20250112 08:56:56.378639 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 48adce00f3644192880a8b16b5e514d9 because the tablet was not found. No further retry: Not found: Tablet not found: 48adce00f3644192880a8b16b5e514d9
I20250112 08:56:56.379871 22456 raft_consensus.cc:357] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.380528 22456 raft_consensus.cc:383] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.380774 22456 raft_consensus.cc:738] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.381598 22456 consensus_queue.cc:260] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.398165 22458 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:56.399624 22458 ts_tablet_manager.cc:1397] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.538s user 0.011s sys 0.001s
I20250112 08:56:56.402470 22458 raft_consensus.cc:357] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.407698 22456 ts_tablet_manager.cc:1428] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.038s user 0.001s sys 0.005s
I20250112 08:56:56.409198 22456 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.418008 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (table default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]) successfully deleted
I20250112 08:56:56.419736 22458 raft_consensus.cc:383] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.423960 22461 tablet_bootstrap.cc:492] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.424412 22461 ts_tablet_manager.cc:1397] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.168s user 0.016s sys 0.019s
I20250112 08:56:56.426607 22461 raft_consensus.cc:357] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.427273 22461 raft_consensus.cc:383] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.427532 22461 raft_consensus.cc:738] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.429137 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0432088806604c248cefd166a4922284"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:56.424485 22458 raft_consensus.cc:738] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.435750 22456 tablet_bootstrap.cc:654] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.438108 22461 consensus_queue.cc:260] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.453655 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 85059ac5ccd844e3893d214ef9ce6b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:56.456442 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet f152b58bd9114691bab99c23b51a5e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:56.458676 22461 ts_tablet_manager.cc:1428] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20250112 08:56:56.465399 22461 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:56.465142 22458 consensus_queue.cc:260] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:56:56.441582 21708 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 27) took 965 ms (client timeout 1000 ms). Trace:
W20250112 08:56:56.466090 21708 rpcz_store.cc:269] 0112 08:56:55.484396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:55.486931 (+ 2535us) service_pool.cc:225] Handling call
0112 08:56:55.575045 (+ 88114us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26572dee 00007fcf242d0fc4 00007fcf242d0f52 00007fcf242d0e81 00007fcf242d0dfc 00007fcf242d00b2 00007fcf2259adfc 00007fcf2259a236
0112 08:56:56.441500 (+866455us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:56.383461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:56.383861 (+ 400us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:56.383884 (+ 23us) write_op.cc:435] Acquired schema lock
0112 08:56:56.383919 (+ 35us) tablet.cc:659] Decoding operations
0112 08:56:56.384165 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:56.384202 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:56.384224 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:56:56.384359 (+ 135us) tablet.cc:698] Row locks acquired
0112 08:56:56.384373 (+ 14us) write_op.cc:260] PREPARE: finished
0112 08:56:56.384476 (+ 103us) write_op.cc:270] Start()
0112 08:56:56.384553 (+ 77us) write_op.cc:276] Timestamp: P: 1736672216384427 usec, L: 0
0112 08:56:56.384567 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:56:56.385039 (+ 472us) log.cc:844] Serialized 532 byte log entry
0112 08:56:56.436874 (+ 51835us) op_driver.cc:464] REPLICATION: finished
0112 08:56:56.438095 (+ 1221us) write_op.cc:301] APPLY: starting
0112 08:56:56.438152 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:56.438237 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:56.438330 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:56.438475 (+ 145us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:56.438624 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:56:56.438664 (+ 40us) write_op.cc:312] APPLY: finished
0112 08:56:56.438826 (+ 162us) log.cc:844] Serialized 22 byte log entry
0112 08:56:56.438969 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:56.439028 (+ 59us) write_op.cc:454] Released schema lock
0112 08:56:56.439057 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 08:56:56.462716 (+ 23659us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:56.462743 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":199011584,"child_traces":[["op",{"apply.queue_time_us":1040,"apply.run_cpu_time_us":1141,"apply.run_wall_time_us":1141,"num_ops":1,"prepare.queue_time_us":735568,"prepare.run_cpu_time_us":2002,"prepare.run_wall_time_us":79239,"raft.queue_time_us":5526,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":636,"replication_time_us":52216}]]}
I20250112 08:56:56.447266 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:56.471983 22461 tablet_bootstrap.cc:654] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.481547 22456 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.481943 22456 ts_tablet_manager.cc:1397] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.007s
I20250112 08:56:56.484342 22456 raft_consensus.cc:357] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.485014 22456 raft_consensus.cc:383] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.485252 22456 raft_consensus.cc:738] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.485792 22456 consensus_queue.cc:260] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.495142 22461 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.495545 22461 ts_tablet_manager.cc:1397] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20250112 08:56:56.498340 22461 raft_consensus.cc:357] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.500309 22461 raft_consensus.cc:383] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.500574 22461 raft_consensus.cc:738] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.501152 22461 consensus_queue.cc:260] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.501519 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fccd34473a6438389cda0b77391c8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.502275 22565 tablet_replica.cc:331] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:56.503006 22565 raft_consensus.cc:2238] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.503438 22565 raft_consensus.cc:2267] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.505259 22565 ts_tablet_manager.cc:1905] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.512087 22461 ts_tablet_manager.cc:1428] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.016s user 0.003s sys 0.005s
I20250112 08:56:56.512112 22458 ts_tablet_manager.cc:1428] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.112s user 0.005s sys 0.000s
I20250112 08:56:56.513375 22456 ts_tablet_manager.cc:1428] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
W20250112 08:56:56.518730 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0432088806604c248cefd166a4922284 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:56.519642 22556 ts_tablet_manager.cc:1918] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:56.524456 22461 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:56.524771 22456 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.529930 22461 tablet_bootstrap.cc:654] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.520126 22556 log.cc:1198] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/745d1a12356244fa811d4b0434ad1295
I20250112 08:56:56.532227 22556 ts_tablet_manager.cc:1939] T 745d1a12356244fa811d4b0434ad1295 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:56.529899 22456 tablet_bootstrap.cc:654] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.561200 22456 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.561592 22456 ts_tablet_manager.cc:1397] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.002s
I20250112 08:56:56.563630 22461 tablet_bootstrap.cc:492] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.564062 22461 ts_tablet_manager.cc:1397] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20250112 08:56:56.566691 22461 raft_consensus.cc:357] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.567366 22461 raft_consensus.cc:383] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 08:56:56.567384 22572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:56:56.568181 22572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:56:56.569015 22572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 08:56:56.568565 22456 raft_consensus.cc:357] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.567626 22461 raft_consensus.cc:738] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.569347 22456 raft_consensus.cc:383] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.569588 22456 raft_consensus.cc:738] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.573546 22461 consensus_queue.cc:260] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.575052 22456 consensus_queue.cc:260] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.585892 22461 ts_tablet_manager.cc:1428] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250112 08:56:56.531137 21708 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:56.588267 22456 ts_tablet_manager.cc:1428] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
W20250112 08:56:56.590478 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a679595edde4a1f9b7f0916b2d760f2 consensus state has no leader
I20250112 08:56:56.592960 22456 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.534765 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 745d1a12356244fa811d4b0434ad1295 (table default.Table_7e088c3591104573aef358a41173f181 [id=66410a0477954e48b2a323036fc388e5]) successfully deleted
I20250112 08:56:56.547210 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 14c7056aaf8844da915acec5f9f92eeb
W20250112 08:56:56.601912 21704 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 67169us
I20250112 08:56:56.602478 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:56.603456 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:56.604166 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 14c7056aaf8844da915acec5f9f92eeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c7056aaf8844da915acec5f9f92eeb already in progress: creating tablet
I20250112 08:56:56.611271 22461 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:56.613360 22456 tablet_bootstrap.cc:654] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.616721 22461 tablet_bootstrap.cc:654] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.414759 22609 raft_consensus.cc:513] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.543131 22565 ts_tablet_manager.cc:1918] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:56.620293 22565 log.cc:1198] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:56.621006 22458 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:56.621052 22565 ts_tablet_manager.cc:1939] T 6fccd34473a6438389cda0b77391c8d7 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:56.602016 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.623072 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 6fccd34473a6438389cda0b77391c8d7 (table default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]) successfully deleted
W20250112 08:56:56.624740 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 14c7056aaf8844da915acec5f9f92eeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c7056aaf8844da915acec5f9f92eeb already in progress: creating tablet
I20250112 08:56:56.628551 22627 tablet.cc:1721] T e456834c46414ae9889e8ff088b79c29 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:56.628677 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:56.632812 22626 tablet.cc:1721] T e456834c46414ae9889e8ff088b79c29 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:56.633713 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:56.634305 22213 raft_consensus.cc:3054] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:56.636354 22609 leader_election.cc:290] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:56:56.638134 22002 leader_election.cc:343] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:56.640062 22572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:56:56.640479 22572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:56:56.640851 22572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:56:56.641167 22572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:56:56.644969 22213 raft_consensus.cc:2463] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:56.646155 21999 leader_election.cc:304] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:56.647401 22609 raft_consensus.cc:2798] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:56.648190 22609 raft_consensus.cc:695] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:56.648999 22609 consensus_queue.cc:237] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.653663 22456 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.654139 22456 ts_tablet_manager.cc:1397] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.002s
I20250112 08:56:56.656805 22456 raft_consensus.cc:357] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:56.657315 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 7451e83f3b554936a79c6724ebe76cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7451e83f3b554936a79c6724ebe76cbb already in progress: creating tablet
I20250112 08:56:56.657719 22456 raft_consensus.cc:383] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.657981 22456 raft_consensus.cc:738] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.658617 22456 consensus_queue.cc:260] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.660498 22556 tablet_replica.cc:331] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:56.661429 22556 raft_consensus.cc:2238] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.662015 22556 raft_consensus.cc:2267] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:56.665596 21704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250112 08:56:56.668452 21704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab -> Peer caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161): Couldn't send request to peer caa33dde8b5248e7ab4d6087d8e9d801. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250112 08:56:56.680956 22556 ts_tablet_manager.cc:1905] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.681156 22456 ts_tablet_manager.cc:1428] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20250112 08:56:56.685225 22458 tablet_bootstrap.cc:654] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.693745 21713 master_service.cc:946] ReplaceTablet: received request to replace tablet c08341b5ce6040d5be30f324d0373e38 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:56.696722 22456 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:56:56.702376 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6182470e64b14e329b278680dc132fcb consensus state has no leader
I20250112 08:56:56.704638 21716 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56650 (request call id 98) took 1001 ms. Trace:
I20250112 08:56:56.704968 21716 rpcz_store.cc:276] 0112 08:56:55.702956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:55.810979 (+108023us) service_pool.cc:225] Handling call
0112 08:56:55.917613 (+106634us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26574306 00007fcf242d8044 00007fcf242d7fd2 00007fcf242d7f01 00007fcf242d7e7c 00007fcf242d7132 00007fcf2259adfc 00007fcf2259a236
0112 08:56:55.919755 (+ 2142us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 08:56:55.927187 (+ 7432us) catalog_manager.cc:2122] Created new table and tablet info
0112 08:56:56.703054 (+775867us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 08:56:56.704336 (+ 1282us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 08:56:56.704581 (+ 245us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:56.651463 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:56.651804 (+ 341us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:56.651824 (+ 20us) write_op.cc:435] Acquired schema lock
0112 08:56:56.651839 (+ 15us) tablet.cc:659] Decoding operations
0112 08:56:56.652196 (+ 357us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:56.652229 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:56.652353 (+ 124us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:56:56.652540 (+ 187us) tablet.cc:698] Row locks acquired
0112 08:56:56.652552 (+ 12us) write_op.cc:260] PREPARE: finished
0112 08:56:56.652661 (+ 109us) write_op.cc:270] Start()
0112 08:56:56.652734 (+ 73us) write_op.cc:276] Timestamp: P: 1736672216652602 usec, L: 0
0112 08:56:56.652748 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:56:56.653168 (+ 420us) log.cc:844] Serialized 432 byte log entry
0112 08:56:56.698476 (+ 45308us) op_driver.cc:464] REPLICATION: finished
0112 08:56:56.698974 (+ 498us) write_op.cc:301] APPLY: starting
0112 08:56:56.699029 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:56.699138 (+ 109us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:56.699264 (+ 126us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:56.699633 (+ 369us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:56.699775 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 08:56:56.699862 (+ 87us) write_op.cc:312] APPLY: finished
0112 08:56:56.700066 (+ 204us) log.cc:844] Serialized 28 byte log entry
0112 08:56:56.700196 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:56.700266 (+ 70us) write_op.cc:454] Released schema lock
0112 08:56:56.702872 (+ 2606us) write_op.cc:341] FINISH: Updating metrics
0112 08:56:56.703487 (+ 615us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:56.703511 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":243115776,"child_traces":[["op",{"apply.queue_time_us":329,"apply.run_cpu_time_us":1540,"apply.run_wall_time_us":4429,"num_ops":2,"prepare.queue_time_us":718594,"prepare.run_cpu_time_us":1717,"prepare.run_wall_time_us":1878,"replication_time_us":45622}]]}
I20250112 08:56:56.712270 22572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:45457,127.21.42.253:41411,127.21.42.252:43161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.42.252
--webserver_port=36567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:56:56.714015 22572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:56:56.696576 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:56.716183 22572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 08:56:56.726471 22456 tablet_bootstrap.cc:654] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.741111 22565 tablet_replica.cc:331] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:56.744376 22565 raft_consensus.cc:2238] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.744871 22565 raft_consensus.cc:2267] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.746168 22456 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.746585 22456 ts_tablet_manager.cc:1397] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.000s
I20250112 08:56:56.749128 22456 raft_consensus.cc:357] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.749799 22456 raft_consensus.cc:383] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.750557 22565 ts_tablet_manager.cc:1905] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:56:56.757829 22637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:56:56.764010 21716 master_service.cc:946] ReplaceTablet: received request to replace tablet a090d258647b4dd7bd91cf64011735dd from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:56.750066 22456 raft_consensus.cc:738] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:56:56.770700 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93be4e0fc31c429f860c1a0cec64849b consensus state has no leader
I20250112 08:56:56.772230 21709 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 85030734daed45d5b67dbffeaa3b8086
I20250112 08:56:56.769714 22456 consensus_queue.cc:260] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.779590 22596 raft_consensus.cc:491] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.780157 22596 raft_consensus.cc:513] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.781075 22456 ts_tablet_manager.cc:1428] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20250112 08:56:56.781360 22458 tablet_bootstrap.cc:492] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:56.781766 22458 ts_tablet_manager.cc:1397] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.003s
I20250112 08:56:56.781960 22596 leader_election.cc:290] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.783927 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:56.784222 22458 raft_consensus.cc:357] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.784525 22213 raft_consensus.cc:2463] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:56.784847 22458 raft_consensus.cc:383] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.785418 22456 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.785926 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:56.790581 22290 leader_election.cc:304] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:56.791122 22556 ts_tablet_manager.cc:1918] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:56:56.791782 22596 raft_consensus.cc:2798] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:56.799659 22596 raft_consensus.cc:491] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:56.800088 22596 raft_consensus.cc:3054] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:56.785075 22458 raft_consensus.cc:738] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.814282 22456 tablet_bootstrap.cc:654] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:56.818660 22596 raft_consensus.cc:513] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.834771 22456 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.835363 22456 ts_tablet_manager.cc:1397] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.001s
I20250112 08:56:56.820977 22556 log.cc:1198] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/48adce00f3644192880a8b16b5e514d9
I20250112 08:56:56.836696 22556 ts_tablet_manager.cc:1939] T 48adce00f3644192880a8b16b5e514d9 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:56.838716 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:56.839318 22213 raft_consensus.cc:3054] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:56.842696 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]) successfully deleted
I20250112 08:56:56.842651 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf912f189a42c19deb744c1c9f4daf" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:56.839919 22596 leader_election.cc:290] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.839408 22456 raft_consensus.cc:357] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.853451 22456 raft_consensus.cc:383] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.853801 22456 raft_consensus.cc:738] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.853289 22556 tablet_replica.cc:331] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:56.854694 22458 consensus_queue.cc:260] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.854586 22456 consensus_queue.cc:260] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.855741 22068 raft_consensus.cc:2463] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:56.855247 22556 raft_consensus.cc:2238] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.857563 22556 raft_consensus.cc:2267] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.861846 22556 ts_tablet_manager.cc:1905] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.872629 22535 raft_consensus.cc:491] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.859766 22456 ts_tablet_manager.cc:1428] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.024s user 0.003s sys 0.007s
I20250112 08:56:56.873049 22535 raft_consensus.cc:513] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.874783 22535 leader_election.cc:290] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.876313 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:56.877825 22146 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.850152 22213 raft_consensus.cc:2463] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:56.879148 22456 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.880187 22290 leader_election.cc:304] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:56:56.882249 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:56.882606 22461 tablet_bootstrap.cc:492] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:56.882951 22461 ts_tablet_manager.cc:1397] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.276s user 0.008s sys 0.004s
I20250112 08:56:56.883772 22596 raft_consensus.cc:2798] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
W20250112 08:56:56.883769 22149 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.884200 22149 leader_election.cc:304] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:56.885344 22535 raft_consensus.cc:2743] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:56.889178 22461 raft_consensus.cc:357] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:56.902472 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6aed47fb6e3d46b6a01ef703d97b3262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:56.904469 22456 tablet_bootstrap.cc:654] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:56.904429 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aed47fb6e3d46b6a01ef703d97b3262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:56.915921 22535 raft_consensus.cc:491] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.916778 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:56.916496 22535 raft_consensus.cc:513] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.917357 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]). Expected version 1 got 0
I20250112 08:56:56.920562 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7451e83f3b554936a79c6724ebe76cbb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:56.921630 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_05ba113b55b64688b747d0552905aab0" } new_table_name: "default.Table_557a9076f797464d99d024ae141d3cc1" modify_external_catalogs: true
W20250112 08:56:56.922214 22146 leader_election.cc:343] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.922835 22535 leader_election.cc:290] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.922961 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:56.923668 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e4f1579fd6744e789018ddc4fbf6a4f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:56.924425 21709 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8dd89a8dac4a4501a9cbe3108faab913" } modify_external_catalogs: true
I20250112 08:56:56.924952 21709 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8dd89a8dac4a4501a9cbe3108faab913" } modify_external_catalogs: true
W20250112 08:56:56.925292 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4f1579fd6744e789018ddc4fbf6a4f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:56.930874 21712 catalog_manager.cc:5526] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } }
I20250112 08:56:56.932958 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]). Expected version 1 got 0
I20250112 08:56:56.935302 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e456834c46414ae9889e8ff088b79c29 (table default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1]). Expected version 1 got 0
I20250112 08:56:56.935690 22456 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:56.936154 22456 ts_tablet_manager.cc:1397] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.000s
I20250112 08:56:56.936432 22535 raft_consensus.cc:491] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:56.936663 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25]). Expected version 2 got 0
I20250112 08:56:56.936872 22535 raft_consensus.cc:513] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.938743 22535 leader_election.cc:290] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:56.940150 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a679595edde4a1f9b7f0916b2d760f2" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:56.938748 22456 raft_consensus.cc:357] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:56.941533 22146 leader_election.cc:343] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:56.941712 22456 raft_consensus.cc:383] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:56.942138 22456 raft_consensus.cc:738] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.951136 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:56.952067 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010ce6f55c444fe0852ad72cc7383cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:56.952193 22456 consensus_queue.cc:260] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:56.957729 22456 ts_tablet_manager.cc:1428] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250112 08:56:56.962225 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:56.962983 22557 tablet_replica.cc:331] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:56.963675 22557 raft_consensus.cc:2238] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:56.964161 22557 raft_consensus.cc:2267] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:56.965950 22557 ts_tablet_manager.cc:1905] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:56.976264 22456 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:56.981556 22041 tablet_service.cc:1467] Processing CreateTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 (DEFAULT_TABLE table=default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:56.984191 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ca0708b2b934445fbd13bbf81e61f61f" } modify_external_catalogs: true
I20250112 08:56:56.984323 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_3fa4cd7715a947e181b264b62212d84c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:56.984706 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ca0708b2b934445fbd13bbf81e61f61f" } modify_external_catalogs: true
W20250112 08:56:56.985868 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fa4cd7715a947e181b264b62212d84c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:56.999135 22456 tablet_bootstrap.cc:654] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.001520 22047 tablet_service.cc:1467] Processing CreateTablet for tablet 574999cd047643d3840a16f2baf6e9bb (DEFAULT_TABLE table=default.Table_05ba113b55b64688b747d0552905aab0 [id=73f85147bf6c40c68cc2a4de785358a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.002261 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 65591a0bb44148c49715b4564dd13524 (DEFAULT_TABLE table=default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.007467 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_96450631653649f3b688e38443936e3f" } new_table_name: "default.Table_bcb5d680dc6047ae9d719a1428694045" modify_external_catalogs: true
I20250112 08:56:56.999172 22192 tablet_service.cc:1467] Processing CreateTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 (DEFAULT_TABLE table=default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.003487 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 574999cd047643d3840a16f2baf6e9bb (DEFAULT_TABLE table=default.Table_05ba113b55b64688b747d0552905aab0 [id=73f85147bf6c40c68cc2a4de785358a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.009109 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88d2f7ac1d4471887c12e9e65e7a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.009758 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574999cd047643d3840a16f2baf6e9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.029192 22458 ts_tablet_manager.cc:1428] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.246s user 0.007s sys 0.001s
I20250112 08:56:57.034889 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010ce6f55c444fe0852ad72cc7383cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.036453 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88d2f7ac1d4471887c12e9e65e7a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.038904 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574999cd047643d3840a16f2baf6e9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.043887 22456 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:57.045387 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65591a0bb44148c49715b4564dd13524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.048708 22535 raft_consensus.cc:491] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.059052 22535 raft_consensus.cc:513] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.060420 22458 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:57.061870 22535 leader_election.cc:290] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.056505 22591 raft_consensus.cc:491] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.062873 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:57.062870 22591 raft_consensus.cc:513] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:57.064146 22146 leader_election.cc:343] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.065318 22591 leader_election.cc:290] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.067138 22456 ts_tablet_manager.cc:1397] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.002s
I20250112 08:56:57.067126 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:57.068387 22146 leader_election.cc:343] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.068658 22557 ts_tablet_manager.cc:1918] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.073055 22557 log.cc:1198] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:56:57.079950 22557 ts_tablet_manager.cc:1939] T 6c683c8c64a441e8b5be52b077fdcb08 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:56.944906 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7451e83f3b554936a79c6724ebe76cbb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:56.946355 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a679595edde4a1f9b7f0916b2d760f2" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:56.950635 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.086895 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 6c683c8c64a441e8b5be52b077fdcb08 (table default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]) successfully deleted
I20250112 08:56:57.087304 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010ce6f55c444fe0852ad72cc7383cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.071656 22456 raft_consensus.cc:357] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.088385 22456 raft_consensus.cc:383] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.088773 22456 raft_consensus.cc:738] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:56.969462 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fccd34473a6438389cda0b77391c8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:56.976570 22336 tablet_service.cc:1467] Processing CreateTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 (DEFAULT_TABLE table=default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.093394 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88d2f7ac1d4471887c12e9e65e7a105. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:56.995200 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 574999cd047643d3840a16f2baf6e9bb (DEFAULT_TABLE table=default.Table_05ba113b55b64688b747d0552905aab0 [id=73f85147bf6c40c68cc2a4de785358a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.096585 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574999cd047643d3840a16f2baf6e9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:56.997416 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 65591a0bb44148c49715b4564dd13524 (DEFAULT_TABLE table=default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.100623 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65591a0bb44148c49715b4564dd13524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.070279 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:57.071727 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:57.090049 22456 consensus_queue.cc:260] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.115144 22456 ts_tablet_manager.cc:1428] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250112 08:56:57.116348 22456 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:57.121250 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet a090d258647b4dd7bd91cf64011735dd
I20250112 08:56:57.122601 21716 catalog_manager.cc:6596] ReplaceTablet: tablet a090d258647b4dd7bd91cf64011735dd deleted and replaced by tablet 267ab3e5955b4d3bbeca44baf021b328
I20250112 08:56:57.123265 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:57.124161 21713 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet c08341b5ce6040d5be30f324d0373e38
I20250112 08:56:57.123984 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.125305 21713 catalog_manager.cc:6596] ReplaceTablet: tablet c08341b5ce6040d5be30f324d0373e38 deleted and replaced by tablet 815d246b7d414aadbb69ac692a5d40c7
I20250112 08:56:57.126686 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:57.127179 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:37418
W20250112 08:56:57.127516 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 574999cd047643d3840a16f2baf6e9bb consensus state has no leader
I20250112 08:56:57.128926 22557 tablet_replica.cc:331] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:57.129489 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:57.130113 22557 raft_consensus.cc:2238] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.130774 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:57.130931 22557 raft_consensus.cc:2267] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:57.134975 22149 leader_election.cc:343] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.135475 22149 leader_election.cc:304] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.136332 22591 raft_consensus.cc:2743] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:57.136711 22149 leader_election.cc:343] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.137190 22149 leader_election.cc:304] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.137987 22591 raft_consensus.cc:2743] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.076004 22596 raft_consensus.cc:695] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:56:57.139302 22596 consensus_queue.cc:237] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.157976 22591 raft_consensus.cc:491] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.158454 22591 raft_consensus.cc:513] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.161861 22556 ts_tablet_manager.cc:1918] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.162453 22556 log.cc:1198] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/85059ac5ccd844e3893d214ef9ce6b9d
I20250112 08:56:57.162752 22591 leader_election.cc:290] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.163153 22556 ts_tablet_manager.cc:1939] T 85059ac5ccd844e3893d214ef9ce6b9d P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:57.165880 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 85059ac5ccd844e3893d214ef9ce6b9d (table default.Table_b959aeabbb59496c93feb6f5fcc5b18c [id=c0068c05bce94c70afa6ea470533e489]) successfully deleted
I20250112 08:56:57.172842 22461 raft_consensus.cc:383] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.173146 22461 raft_consensus.cc:738] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:57.173856 22461 consensus_queue.cc:260] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.184196 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 65591a0bb44148c49715b4564dd13524 (DEFAULT_TABLE table=default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.185024 22332 tablet_service.cc:1467] Processing CreateTablet for tablet da0c101de2544cc9907c78fdf1a4af82 (DEFAULT_TABLE table=default.Table_5ca837d5572348a9841f503e80c7f126 [id=5a0803ed62d544138a3ea814b46f665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.185324 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65591a0bb44148c49715b4564dd13524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.185112 22046 tablet_service.cc:1467] Processing CreateTablet for tablet da0c101de2544cc9907c78fdf1a4af82 (DEFAULT_TABLE table=default.Table_5ca837d5572348a9841f503e80c7f126 [id=5a0803ed62d544138a3ea814b46f665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.186050 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da0c101de2544cc9907c78fdf1a4af82. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:57.192211 22429 log.cc:926] Time spent T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Append to log took a long time: real 0.084s user 0.007s sys 0.007s
I20250112 08:56:57.195220 22556 tablet_replica.cc:331] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:57.196161 22556 raft_consensus.cc:2238] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.196714 22556 raft_consensus.cc:2267] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:57.138384 22149 leader_election.cc:343] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.212368 22149 leader_election.cc:304] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.213229 22591 raft_consensus.cc:2743] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:57.214228 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 75875.7us
W20250112 08:56:57.214910 22149 leader_election.cc:343] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.215329 22149 leader_election.cc:304] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.219048 22557 ts_tablet_manager.cc:1905] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.222366 22556 ts_tablet_manager.cc:1905] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.224687 22565 ts_tablet_manager.cc:1918] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.225111 22565 log.cc:1198] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:57.225828 22565 ts_tablet_manager.cc:1939] T 69a9c2b5a5d044c789a3eda07e8a312d P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:57.228993 22644 raft_consensus.cc:491] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.229444 22644 raft_consensus.cc:513] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.233398 22456 tablet_bootstrap.cc:654] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:57.233301 22290 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:57.246835 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 93be4e0fc31c429f860c1a0cec64849b
I20250112 08:56:57.249143 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:57.258610 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 65591a0bb44148c49715b4564dd13524
I20250112 08:56:57.261536 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:57.268438 22461 ts_tablet_manager.cc:1428] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.385s user 0.003s sys 0.005s
I20250112 08:56:57.269307 22461 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:57.270471 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_c18e0bb841d942978709c4d445929483"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.271991 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c18e0bb841d942978709c4d445929483 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.274521 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08341b5ce6040d5be30f324d0373e38" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:57.280434 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da0c101de2544cc9907c78fdf1a4af82. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:57.284735 22290 leader_election.cc:343] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:57.285161 22290 leader_election.cc:304] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:57.250233 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.260182 22190 tablet_service.cc:1467] Processing CreateTablet for tablet da0c101de2544cc9907c78fdf1a4af82 (DEFAULT_TABLE table=default.Table_5ca837d5572348a9841f503e80c7f126 [id=5a0803ed62d544138a3ea814b46f665b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:57.306458 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da0c101de2544cc9907c78fdf1a4af82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:57.310685 21716 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]). Expected version 1 got 0
I20250112 08:56:57.311424 21716 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]). Expected version 1 got 0
I20250112 08:56:57.262136 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.266800 22557 ts_tablet_manager.cc:1918] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.312285 22557 log.cc:1198] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/a090d258647b4dd7bd91cf64011735dd
I20250112 08:56:57.312922 22557 ts_tablet_manager.cc:1939] T a090d258647b4dd7bd91cf64011735dd P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:56:57.276497 22146 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.322211 21711 catalog_manager.cc:5526] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:57.323791 22596 raft_consensus.cc:2743] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.325215 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_e86edc5a626a48a195a3a19d05113984 [id=f69f93bdf75d4f1fb836d8961cbc68c4]). Expected version 1 got 0
I20250112 08:56:57.335230 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 69651ab160db41b3a187542ac1cd2839 (table default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]). Expected version 1 got 0
W20250112 08:56:57.336277 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 422e1484cd84479abba824cc8a6fae1f has no consensus state
I20250112 08:56:57.345471 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_6aed47fb6e3d46b6a01ef703d97b3262" } new_table_name: "default.Table_8eaaa4847a874dd19d4bda6a4d4fd632" modify_external_catalogs: true
I20250112 08:56:57.370703 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_89c2768382e2459894ea3add3ee2319c [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 2 got 0
W20250112 08:56:57.375061 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 90362.8us
I20250112 08:56:57.345387 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_51785214aefc4cb0abcc3983ff977c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.378597 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51785214aefc4cb0abcc3983ff977c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.379200 22653 raft_consensus.cc:491] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.379627 22653 raft_consensus.cc:513] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.355437 21712 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c683c8c64a441e8b5be52b077fdcb08 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:57.367061 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet a090d258647b4dd7bd91cf64011735dd from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:57.392015 22644 leader_election.cc:290] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.397733 22591 raft_consensus.cc:2743] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:57.398492 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 183608us
I20250112 08:56:57.400197 22535 raft_consensus.cc:491] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.400082 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08341b5ce6040d5be30f324d0373e38" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:57.400935 22557 tablet_replica.cc:331] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:57.400592 22535 raft_consensus.cc:513] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.401604 22557 raft_consensus.cc:2238] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.402014 22557 raft_consensus.cc:2267] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:57.402280 22535 leader_election.cc:290] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.403254 22535 raft_consensus.cc:491] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.403470 22557 ts_tablet_manager.cc:1905] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.405850 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c7056aaf8844da915acec5f9f92eeb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:57.408176 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ad9b1593a24424989f380811954439e has no consensus state
W20250112 08:56:57.408787 22146 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.409871 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet a090d258647b4dd7bd91cf64011735dd (table default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
I20250112 08:56:57.403645 22535 raft_consensus.cc:513] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.413725 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:57.413722 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:57.414040 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:57.415941 21716 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_c0a3687f36764454a96695c068067569 [id=70b6c08dd74f4211ad7034a109e70eb1] alter complete (version 1)
W20250112 08:56:57.415145 22146 leader_election.cc:343] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.418517 22458 tablet_bootstrap.cc:700] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Time spent opening tablet: real 0.199s user 0.002s sys 0.000s
I20250112 08:56:57.418347 22535 leader_election.cc:290] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.419111 22458 tablet_bootstrap.cc:654] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.438313 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:57.441334 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.443212 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e456834c46414ae9889e8ff088b79c29" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:57.445013 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b75895a7c9040c1a2ce8c25aeb5c32b with delete_type TABLET_DATA_DELETED (replaced by tablet 85030734daed45d5b67dbffeaa3b8086) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:57.446990 22290 leader_election.cc:304] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:57.448902 22644 raft_consensus.cc:2743] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.455313 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_bd96331208304ec68d07635be2602337"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.456815 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd96331208304ec68d07635be2602337 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.457428 21711 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_94180f60c7fb492eb9658711bc6c492d" } modify_external_catalogs: true
I20250112 08:56:57.457957 21711 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_94180f60c7fb492eb9658711bc6c492d" } modify_external_catalogs: true
I20250112 08:56:57.462579 22653 leader_election.cc:290] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.462931 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_e4f1579fd6744e789018ddc4fbf6a4f6" } modify_external_catalogs: true
I20250112 08:56:57.463354 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_e4f1579fd6744e789018ddc4fbf6a4f6" } modify_external_catalogs: true
I20250112 08:56:57.473261 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_97d2a91e67624174904c1f044679de1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.474759 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d2a91e67624174904c1f044679de1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.491252 22461 tablet_bootstrap.cc:654] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.492674 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 69a9c2b5a5d044c789a3eda07e8a312d (table default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]) successfully deleted
W20250112 08:56:57.493347 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet a090d258647b4dd7bd91cf64011735dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a090d258647b4dd7bd91cf64011735dd already in progress: creating tablet
W20250112 08:56:57.494113 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet c08341b5ce6040d5be30f324d0373e38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c08341b5ce6040d5be30f324d0373e38 already in progress: creating tablet
W20250112 08:56:57.494669 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 93be4e0fc31c429f860c1a0cec64849b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93be4e0fc31c429f860c1a0cec64849b already in progress: creating tablet
W20250112 08:56:57.495182 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 65591a0bb44148c49715b4564dd13524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65591a0bb44148c49715b4564dd13524 already in progress: creating tablet
I20250112 08:56:57.516983 22458 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:57.517383 22458 ts_tablet_manager.cc:1397] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.457s user 0.008s sys 0.001s
I20250112 08:56:57.522538 21714 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1f8ccbe0a92f4bc9be3fa326b9cd47ec
I20250112 08:56:57.533083 22458 raft_consensus.cc:357] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.533926 22458 raft_consensus.cc:383] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.534235 22458 raft_consensus.cc:738] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:57.534974 22458 consensus_queue.cc:260] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.544677 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1acedaae5cd2449d9d9cbc5a91994176"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.546149 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1acedaae5cd2449d9d9cbc5a91994176 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.560858 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
W20250112 08:56:57.563408 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet f252f0ec641d4bf5b4fe1399a8ced51e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f252f0ec641d4bf5b4fe1399a8ced51e already in progress: creating tablet
W20250112 08:56:57.563724 22002 consensus_peers.cc:487] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:57.570019 22653 raft_consensus.cc:491] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.570605 22653 raft_consensus.cc:513] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.574709 22653 leader_election.cc:290] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:56:57.575217 22290 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:57.576399 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c7056aaf8844da915acec5f9f92eeb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:57.580229 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:57.582623 22149 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.583709 22149 leader_election.cc:304] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:57.585428 22149 raft_consensus.cc:2681] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 08:56:57.586782 22149 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.588104 22149 leader_election.cc:304] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:57.593986 22290 leader_election.cc:343] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:57.594657 22290 leader_election.cc:304] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:57.594990 21711 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8f3c071a32c84bc0b9894e2f82d92b97
I20250112 08:56:57.596065 22653 raft_consensus.cc:2743] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.589684 22192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.600442 22556 ts_tablet_manager.cc:1918] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.590655 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:57.599612 22213 raft_consensus.cc:1270] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Refusing update from remote peer 76718d1596134c83900541c6d398d59e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:57.600865 22556 log.cc:1198] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/f152b58bd9114691bab99c23b51a5e89
I20250112 08:56:57.601545 22556 ts_tablet_manager.cc:1939] T f152b58bd9114691bab99c23b51a5e89 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:57.604272 22609 consensus_queue.cc:1035] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:57.611828 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 61) took 1149 ms. Trace:
I20250112 08:56:57.612331 22556 rpcz_store.cc:276] 0112 08:56:56.454761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:56.456283 (+ 1522us) service_pool.cc:225] Handling call
0112 08:56:57.195040 (+738757us) ts_tablet_manager.cc:1145] Deleting tablet f152b58bd9114691bab99c23b51a5e89
0112 08:56:57.195089 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:57.238829 (+ 43740us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:57.600178 (+361349us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:57.603858 (+ 3680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":738063}
I20250112 08:56:57.614004 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 62) took 1011 ms. Trace:
I20250112 08:56:57.614401 22556 rpcz_store.cc:276] 0112 08:56:56.603072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:56.603308 (+ 236us) service_pool.cc:225] Handling call
0112 08:56:57.613633 (+1010325us) ts_tablet_manager.cc:1145] Deleting tablet 14c7056aaf8844da915acec5f9f92eeb
0112 08:56:57.613678 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:57.613948 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009738}
I20250112 08:56:57.615564 22556 tablet_replica.cc:331] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:57.616434 22556 raft_consensus.cc:2238] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.617111 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet f152b58bd9114691bab99c23b51a5e89 (table default.Table_b31a7de582934a729d5a6070c4d7fb32 [id=6ae86a04705444a6ba14742d84959e6d]) successfully deleted
I20250112 08:56:57.617420 22556 raft_consensus.cc:2267] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:57.617935 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 14c7056aaf8844da915acec5f9f92eeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c7056aaf8844da915acec5f9f92eeb already in progress: creating tablet
I20250112 08:56:57.621337 22556 ts_tablet_manager.cc:1905] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.622198 22591 raft_consensus.cc:2743] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.628173 22456 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:57.628057 21711 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_e86edc5a626a48a195a3a19d05113984" } new_table_name: "default.Table_7971251621ad4bb69351f4beb52fa190" modify_external_catalogs: true
I20250112 08:56:57.628758 22456 ts_tablet_manager.cc:1397] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.513s user 0.009s sys 0.008s
I20250112 08:56:57.636632 22456 raft_consensus.cc:357] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.638666 22456 raft_consensus.cc:383] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.638978 22456 raft_consensus.cc:738] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:56:57.641009 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a679595edde4a1f9b7f0916b2d760f2 consensus state has no leader
I20250112 08:56:57.643693 22461 tablet_bootstrap.cc:492] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:57.648269 22461 ts_tablet_manager.cc:1397] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.379s user 0.006s sys 0.004s
I20250112 08:56:57.650620 22461 raft_consensus.cc:357] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.651398 22461 raft_consensus.cc:383] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.653162 22458 ts_tablet_manager.cc:1428] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.136s user 0.003s sys 0.005s
I20250112 08:56:57.658375 22456 consensus_queue.cc:260] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:57.645689 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 58849us
I20250112 08:56:57.660208 22458 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:56:57.660485 22149 leader_election.cc:343] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.661010 22149 leader_election.cc:304] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.667389 21716 master_service.cc:946] ReplaceTablet: received request to replace tablet b0192d66e64d4d39b4bcfd6565dff723 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:57.671164 22461 raft_consensus.cc:738] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:57.672776 21712 master_service.cc:946] ReplaceTablet: received request to replace tablet dabc37f08d1948118b98ba25b689bea4 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:57.681198 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_62b29cf7e29a49a587abdbe532a560c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:57.683059 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62b29cf7e29a49a587abdbe532a560c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.683702 22461 consensus_queue.cc:260] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.696870 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_89c2768382e2459894ea3add3ee2319c" } new_table_name: "default.Table_0cd1e966451c4f1c9b258c3053f4a337" modify_external_catalogs: true
I20250112 08:56:57.705029 22458 tablet_bootstrap.cc:654] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.710139 22662 raft_consensus.cc:2743] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.711633 22557 ts_tablet_manager.cc:1918] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:57.712061 22557 log.cc:1198] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/c08341b5ce6040d5be30f324d0373e38
W20250112 08:56:57.714208 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 53798us
I20250112 08:56:57.729282 22355 raft_consensus.cc:3054] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:57.716034 22456 ts_tablet_manager.cc:1428] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.087s user 0.007s sys 0.004s
I20250112 08:56:57.723946 22557 ts_tablet_manager.cc:1939] T c08341b5ce6040d5be30f324d0373e38 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:57.749639 22456 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:57.756558 22667 tablet.cc:1721] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:57.762346 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:57.764128 22653 raft_consensus.cc:491] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.764397 22565 tablet_replica.cc:331] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:57.765089 22565 raft_consensus.cc:2238] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.765512 22565 raft_consensus.cc:2267] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:57.764758 22653 raft_consensus.cc:513] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.768827 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:57.769346 22565 ts_tablet_manager.cc:1905] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.770943 22356 raft_consensus.cc:3054] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:57.773308 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa641fe365844c9eaa6ffcbf548d82f7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:57.773658 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b8ef9dfb2cb1497bb83b09e8bff95c68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:57.774682 22290 leader_election.cc:304] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:57.775593 22653 leader_election.cc:290] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.776701 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_a2e98237a7f34d58875c6af5ab558a33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:57.779625 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bd96331208304ec68d07635be2602337" } modify_external_catalogs: true
I20250112 08:56:57.780313 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bd96331208304ec68d07635be2602337" } modify_external_catalogs: true
I20250112 08:56:57.789367 22068 raft_consensus.cc:3054] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:57.799512 22461 ts_tablet_manager.cc:1428] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.151s user 0.006s sys 0.001s
I20250112 08:56:57.800604 22461 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:56:57.806566 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8ef9dfb2cb1497bb83b09e8bff95c68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.807492 22557 tablet_replica.cc:331] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: stopping tablet replica
W20250112 08:56:57.803182 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6182470e64b14e329b278680dc132fcb consensus state has no leader
I20250112 08:56:57.815668 22068 raft_consensus.cc:1270] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Refusing update from remote peer 8c608785d6874abf9cc28db11a979ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:57.817293 22653 consensus_queue.cc:1035] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:56:57.818553 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]). Expected version 1 got 0
I20250112 08:56:57.831851 22557 raft_consensus.cc:2238] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:57.841607 21708 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 56e990ca6fc0461f91515536ea477665
I20250112 08:56:57.843309 22557 raft_consensus.cc:2267] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:57.845973 22557 ts_tablet_manager.cc:1905] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:57.832787 22668 tablet.cc:1721] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:57.833536 22662 raft_consensus.cc:491] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:57.847445 22662 raft_consensus.cc:513] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.852018 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:57.854158 22146 leader_election.cc:343] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.854956 22662 leader_election.cc:290] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:57.857229 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:57.859329 22149 leader_election.cc:343] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:57.860080 22149 leader_election.cc:304] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:57.862777 22662 raft_consensus.cc:2743] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:57.804319 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet c08341b5ce6040d5be30f324d0373e38 (table default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]) successfully deleted
W20250112 08:56:57.871852 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 67564.4us
W20250112 08:56:57.872819 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 93be4e0fc31c429f860c1a0cec64849b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93be4e0fc31c429f860c1a0cec64849b already in progress: creating tablet
W20250112 08:56:57.873744 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 65591a0bb44148c49715b4564dd13524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65591a0bb44148c49715b4564dd13524 already in progress: creating tablet
I20250112 08:56:57.889107 22674 tablet.cc:1721] T e456834c46414ae9889e8ff088b79c29 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:57.890794 21715 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6] alter complete (version 1)
I20250112 08:56:57.891736 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3f79f09bbc3949a68c55ec5dab776291" } modify_external_catalogs: true
I20250112 08:56:57.892253 22458 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:57.892644 22458 ts_tablet_manager.cc:1397] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.233s user 0.009s sys 0.001s
I20250112 08:56:57.892804 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3f79f09bbc3949a68c55ec5dab776291" } modify_external_catalogs: true
I20250112 08:56:57.899178 22461 tablet_bootstrap.cc:654] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:57.904630 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2e98237a7f34d58875c6af5ab558a33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.839442 22456 tablet_bootstrap.cc:654] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.840261 22212 raft_consensus.cc:1270] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Refusing update from remote peer 8c608785d6874abf9cc28db11a979ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:56:57.909900 22653 consensus_queue.cc:1035] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:56:57.895673 22458 raft_consensus.cc:357] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.953122 22458 raft_consensus.cc:383] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.953424 22458 raft_consensus.cc:738] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:57.956037 22667 tablet.cc:1721] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:57.960208 22458 consensus_queue.cc:260] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:57.966156 22458 ts_tablet_manager.cc:1428] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20250112 08:56:57.967859 22458 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:57.980736 22677 tablet.cc:1721] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:57.983585 22458 tablet_bootstrap.cc:654] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:57.986261 22461 tablet_bootstrap.cc:492] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:57.987066 22461 ts_tablet_manager.cc:1397] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.187s user 0.010s sys 0.005s
I20250112 08:56:57.989315 22461 raft_consensus.cc:357] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:57.990075 22461 raft_consensus.cc:383] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:57.990482 22461 raft_consensus.cc:738] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:57.994755 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_39462b870602435cb6f626b7cbf102f6" } new_table_name: "default.Table_00d2c67bdfbd47a6a03b2fb22f72f193" modify_external_catalogs: true
I20250112 08:56:57.999202 21715 master_service.cc:946] ReplaceTablet: received request to replace tablet aa641fe365844c9eaa6ffcbf548d82f7 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:58.014658 22556 ts_tablet_manager.cc:1918] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:58.015045 22556 log.cc:1198] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/6fccd34473a6438389cda0b77391c8d7
I20250112 08:56:58.015728 22556 ts_tablet_manager.cc:1939] T 6fccd34473a6438389cda0b77391c8d7 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:58.019001 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 64) took 1049 ms. Trace:
I20250112 08:56:58.019443 22556 rpcz_store.cc:276] 0112 08:56:56.969205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:56.969336 (+ 131us) service_pool.cc:225] Handling call
0112 08:56:57.615399 (+646063us) ts_tablet_manager.cc:1145] Deleting tablet 6fccd34473a6438389cda0b77391c8d7
0112 08:56:57.615442 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:57.911349 (+295907us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.014408 (+103059us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.018940 (+ 4532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":522704}
I20250112 08:56:58.020406 22683 tablet.cc:1721] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:56:58.032232 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 6fccd34473a6438389cda0b77391c8d7 (table default.Table_7f4572b6e0aa4803b91c2ad6bbab4ec3 [id=7bc4b38f01534c74824e94e411ba2935]) successfully deleted
W20250112 08:56:58.033210 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet c08341b5ce6040d5be30f324d0373e38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c08341b5ce6040d5be30f324d0373e38 already in progress: creating tablet
W20250112 08:56:58.033962 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet a090d258647b4dd7bd91cf64011735dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a090d258647b4dd7bd91cf64011735dd already in progress: creating tablet
W20250112 08:56:58.034797 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 93be4e0fc31c429f860c1a0cec64849b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93be4e0fc31c429f860c1a0cec64849b already in progress: creating tablet
I20250112 08:56:58.036465 22556 tablet_replica.cc:331] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:58.037312 22556 raft_consensus.cc:2238] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:58.037770 22556 raft_consensus.cc:2267] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:56:58.039240 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 65591a0bb44148c49715b4564dd13524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65591a0bb44148c49715b4564dd13524 already in progress: creating tablet
I20250112 08:56:58.048311 22556 ts_tablet_manager.cc:1905] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.060792 22685 tablet.cc:1721] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:56:58.063086 22456 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:58.063606 22456 ts_tablet_manager.cc:1397] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.314s user 0.012s sys 0.004s
I20250112 08:56:58.068888 22644 raft_consensus.cc:2743] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.089413 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_bda308e111924297bc1d064e44647a8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:58.091995 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_abb85608f59e40c1bcb4004eb5783cb6" } modify_external_catalogs: true
I20250112 08:56:58.092537 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_abb85608f59e40c1bcb4004eb5783cb6" } modify_external_catalogs: true
W20250112 08:56:58.093502 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bda308e111924297bc1d064e44647a8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:58.097417 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_da91c42fe80845d5b255227302fa7244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:58.099589 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da91c42fe80845d5b255227302fa7244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:57.991169 22461 consensus_queue.cc:260] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:58.105269 22644 raft_consensus.cc:491] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.105731 22644 raft_consensus.cc:513] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:56:58.110313 22644 leader_election.cc:290] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:56:58.124747 22290 leader_election.cc:343] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:58.130895 22456 raft_consensus.cc:357] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.131588 22456 raft_consensus.cc:383] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.131433 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 06717cb567a54ee6a268c3be5d8786ec (DEFAULT_TABLE table=default.Table_c18e0bb841d942978709c4d445929483 [id=4e7fa7e2c76f4932bd3ae514033670e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.131842 22456 raft_consensus.cc:738] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.132418 22456 consensus_queue.cc:260] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.137764 22677 tablet.cc:1716] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:58.138185 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:56:58.139430 21715 catalog_manager.cc:6596] ReplaceTablet: tablet aa641fe365844c9eaa6ffcbf548d82f7 deleted and replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:56:58.140885 22335 tablet_service.cc:1467] Processing CreateTablet for tablet e19979ef29944405b833bd6dad6f3664 (DEFAULT_TABLE table=default.Table_3fa4cd7715a947e181b264b62212d84c [id=d999ba318b994d1a9bf35f14a004d1bf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:58.141600 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a72d2062bf2a4c79b5a8be974d1b8cb5 consensus state has no leader
I20250112 08:56:58.143523 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:58.144654 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9b1593a24424989f380811954439e (DEFAULT_TABLE table=default.Table_8eaaa4847a874dd19d4bda6a4d4fd632 [id=650e7fc35dcf4384a76a47c2d6f24665]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:58.150454 22290 leader_election.cc:343] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:56:58.150866 22290 leader_election.cc:304] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:58.152056 22456 ts_tablet_manager.cc:1428] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.088s user 0.010s sys 0.001s
I20250112 08:56:58.156607 22456 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:58.158128 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3fa4cd7715a947e181b264b62212d84c" } new_table_name: "default.Table_cc848572e4f74f148ed5aa8bb6f16da3" modify_external_catalogs: true
I20250112 08:56:58.160318 22683 tablet.cc:1716] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:58.163214 22681 raft_consensus.cc:2743] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.170789 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 35f60d1f4bed4721b73deb9ba787417c (DEFAULT_TABLE table=default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.173151 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 422e1484cd84479abba824cc8a6fae1f (DEFAULT_TABLE table=default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.174916 22456 tablet_bootstrap.cc:654] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.016796 22565 ts_tablet_manager.cc:1918] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:58.182875 22565 log.cc:1198] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:56:58.183552 22565 ts_tablet_manager.cc:1939] T f252f0ec641d4bf5b4fe1399a8ced51e P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:58.183635 22333 tablet_service.cc:1467] Processing CreateTablet for tablet a5959caa4d39420981e3b1994df4cb17 (DEFAULT_TABLE table=default.Table_7111dd9cee36485ab448639684e10659 [id=8172f1c5f3b54191aaa2322ccd8c638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.145243 22667 tablet.cc:1716] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Skipping requested alter to schema version 2, tablet already version 2
I20250112 08:56:58.175626 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 06717cb567a54ee6a268c3be5d8786ec (DEFAULT_TABLE table=default.Table_c18e0bb841d942978709c4d445929483 [id=4e7fa7e2c76f4932bd3ae514033670e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.187862 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06717cb567a54ee6a268c3be5d8786ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.206869 22456 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:58.207402 22456 ts_tablet_manager.cc:1397] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.007s
I20250112 08:56:58.208552 22461 ts_tablet_manager.cc:1428] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.221s user 0.005s sys 0.000s
I20250112 08:56:58.215775 22456 raft_consensus.cc:357] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.216711 22456 raft_consensus.cc:383] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.216953 22456 raft_consensus.cc:738] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.222182 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06717cb567a54ee6a268c3be5d8786ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.223676 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19979ef29944405b833bd6dad6f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.225031 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9b1593a24424989f380811954439e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.226255 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422e1484cd84479abba824cc8a6fae1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.227283 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5959caa4d39420981e3b1994df4cb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.232450 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f60d1f4bed4721b73deb9ba787417c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.240196 22456 consensus_queue.cc:260] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.260282 22456 ts_tablet_manager.cc:1428] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.052s user 0.008s sys 0.000s
I20250112 08:56:58.271764 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 06717cb567a54ee6a268c3be5d8786ec (DEFAULT_TABLE table=default.Table_c18e0bb841d942978709c4d445929483 [id=4e7fa7e2c76f4932bd3ae514033670e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.274101 22046 tablet_service.cc:1467] Processing CreateTablet for tablet e19979ef29944405b833bd6dad6f3664 (DEFAULT_TABLE table=default.Table_3fa4cd7715a947e181b264b62212d84c [id=d999ba318b994d1a9bf35f14a004d1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.275306 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19979ef29944405b833bd6dad6f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.278338 22456 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:58.285454 22192 tablet_service.cc:1467] Processing CreateTablet for tablet e19979ef29944405b833bd6dad6f3664 (DEFAULT_TABLE table=default.Table_3fa4cd7715a947e181b264b62212d84c [id=d999ba318b994d1a9bf35f14a004d1bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.287770 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9b1593a24424989f380811954439e (DEFAULT_TABLE table=default.Table_8eaaa4847a874dd19d4bda6a4d4fd632 [id=650e7fc35dcf4384a76a47c2d6f24665]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.288858 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:58.289196 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9b1593a24424989f380811954439e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.290359 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:58.291127 22565 tablet_replica.cc:331] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:58.291812 22565 raft_consensus.cc:2238] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:56:58.292692 22565 raft_consensus.cc:2267] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:58.294907 22565 ts_tablet_manager.cc:1905] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.294907 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad9b1593a24424989f380811954439e (DEFAULT_TABLE table=default.Table_8eaaa4847a874dd19d4bda6a4d4fd632 [id=650e7fc35dcf4384a76a47c2d6f24665]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.302505 22461 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:58.319491 22456 tablet_bootstrap.cc:654] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.305588 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 815d246b7d414aadbb69ac692a5d40c7 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.306093 22331 tablet_service.cc:1467] Processing CreateTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 (DEFAULT_TABLE table=default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.308007 22330 tablet_service.cc:1467] Processing CreateTablet for tablet c9848b879a2e40aeb840924261e6a100 (DEFAULT_TABLE table=default.Table_d3832ac137d5425db9a0f35eb56cfd68 [id=6fa448eed39749ffa607e4ca225e4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.309556 22329 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1c634513e14e818601894834c41378 (DEFAULT_TABLE table=default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.350817 22456 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:58.351398 22456 ts_tablet_manager.cc:1397] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.000s
I20250112 08:56:58.354627 22456 raft_consensus.cc:357] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.361821 22047 tablet_service.cc:1467] Processing CreateTablet for tablet 35f60d1f4bed4721b73deb9ba787417c (DEFAULT_TABLE table=default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.364199 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 35f60d1f4bed4721b73deb9ba787417c (DEFAULT_TABLE table=default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.355643 22456 raft_consensus.cc:383] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.366079 22456 raft_consensus.cc:738] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.367429 22456 consensus_queue.cc:260] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.368304 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 422e1484cd84479abba824cc8a6fae1f (DEFAULT_TABLE table=default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.370668 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 422e1484cd84479abba824cc8a6fae1f (DEFAULT_TABLE table=default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.376286 22456 ts_tablet_manager.cc:1428] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20250112 08:56:58.377454 22662 raft_consensus.cc:491] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.380604 22662 raft_consensus.cc:513] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.382771 22458 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:58.383163 22458 ts_tablet_manager.cc:1397] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.415s user 0.007s sys 0.002s
I20250112 08:56:58.383308 22662 leader_election.cc:290] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:58.385478 22458 raft_consensus.cc:357] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.386164 22458 raft_consensus.cc:383] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.386410 22458 raft_consensus.cc:738] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.386670 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:58.388322 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:58.390655 22146 leader_election.cc:343] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:58.392102 22149 leader_election.cc:343] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.392506 22149 leader_election.cc:304] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:58.395222 22662 raft_consensus.cc:2743] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.397720 22456 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:58.409154 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 6ffbab945abd4fa58259a65e4fadd622
I20250112 08:56:58.411350 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:58.413429 22328 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:58 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:58.413673 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:58.417971 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_250ba1babd9c419e8933849397cdff93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:58.419008 22040 tablet_service.cc:1467] Processing CreateTablet for tablet a5959caa4d39420981e3b1994df4cb17 (DEFAULT_TABLE table=default.Table_7111dd9cee36485ab448639684e10659 [id=8172f1c5f3b54191aaa2322ccd8c638d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:58.419530 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_250ba1babd9c419e8933849397cdff93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:58.420825 22188 tablet_service.cc:1467] Processing CreateTablet for tablet a5959caa4d39420981e3b1994df4cb17 (DEFAULT_TABLE table=default.Table_7111dd9cee36485ab448639684e10659 [id=8172f1c5f3b54191aaa2322ccd8c638d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.422675 22187 tablet_service.cc:1467] Processing CreateTablet for tablet 815d246b7d414aadbb69ac692a5d40c7 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.421677 22456 tablet_bootstrap.cc:654] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.424432 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 (DEFAULT_TABLE table=default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.426126 22185 tablet_service.cc:1467] Processing CreateTablet for tablet c9848b879a2e40aeb840924261e6a100 (DEFAULT_TABLE table=default.Table_d3832ac137d5425db9a0f35eb56cfd68 [id=6fa448eed39749ffa607e4ca225e4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.427678 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1c634513e14e818601894834c41378 (DEFAULT_TABLE table=default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.429212 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422e1484cd84479abba824cc8a6fae1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.432335 22565 ts_tablet_manager.cc:1918] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:56:58.432699 22565 log.cc:1198] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:56:58.433360 22565 ts_tablet_manager.cc:1939] T aa641fe365844c9eaa6ffcbf548d82f7 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:58.434128 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06717cb567a54ee6a268c3be5d8786ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.435529 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19979ef29944405b833bd6dad6f3664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.437489 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad9b1593a24424989f380811954439e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.438637 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f60d1f4bed4721b73deb9ba787417c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.443207 22456 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:58.443759 22456 ts_tablet_manager.cc:1397] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.000s
I20250112 08:56:58.444248 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422e1484cd84479abba824cc8a6fae1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.445626 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f60d1f4bed4721b73deb9ba787417c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.446942 22456 raft_consensus.cc:357] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.457223 22557 ts_tablet_manager.cc:1918] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:58.457630 22557 log.cc:1198] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/14c7056aaf8844da915acec5f9f92eeb
I20250112 08:56:58.458271 22557 ts_tablet_manager.cc:1939] T 14c7056aaf8844da915acec5f9f92eeb P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:56:58.460990 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 (DEFAULT_TABLE table=default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.461488 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815d246b7d414aadbb69ac692a5d40c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.462677 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1c634513e14e818601894834c41378 (DEFAULT_TABLE table=default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.463789 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5959caa4d39420981e3b1994df4cb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.464743 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ab3e5955b4d3bbeca44baf021b328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.465720 22461 tablet_bootstrap.cc:654] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.466641 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9848b879a2e40aeb840924261e6a100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.467563 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815d246b7d414aadbb69ac692a5d40c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.468374 22329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1c634513e14e818601894834c41378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.461030 22042 tablet_service.cc:1467] Processing CreateTablet for tablet c9848b879a2e40aeb840924261e6a100 (DEFAULT_TABLE table=default.Table_d3832ac137d5425db9a0f35eb56cfd68 [id=6fa448eed39749ffa607e4ca225e4fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.476292 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1c634513e14e818601894834c41378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.476961 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9848b879a2e40aeb840924261e6a100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.478304 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ab3e5955b4d3bbeca44baf021b328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.386940 22458 consensus_queue.cc:260] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.481405 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5959caa4d39420981e3b1994df4cb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.486084 22458 ts_tablet_manager.cc:1428] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.103s user 0.001s sys 0.005s
I20250112 08:56:58.487820 22662 raft_consensus.cc:491] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.488273 22662 raft_consensus.cc:513] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.490130 22662 leader_election.cc:290] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:58.491117 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:58.492458 22146 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.496847 22183 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:58 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:58.499557 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:58 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:58.509176 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet f252f0ec641d4bf5b4fe1399a8ced51e (table default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]) successfully deleted
I20250112 08:56:58.512939 22458 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:58.513716 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 81) took 1073 ms. Trace:
I20250112 08:56:58.514025 22557 rpcz_store.cc:276] 0112 08:56:57.440015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:57.441148 (+ 1133us) service_pool.cc:225] Handling call
0112 08:56:57.807271 (+366123us) ts_tablet_manager.cc:1145] Deleting tablet 14c7056aaf8844da915acec5f9f92eeb
0112 08:56:57.807341 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:58.225980 (+418639us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.428752 (+202772us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.513647 (+ 84895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":365517}
I20250112 08:56:58.515237 22557 tablet_replica.cc:331] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:58.530418 22565 tablet_replica.cc:331] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:58.560932 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:58.574965 22565 raft_consensus.cc:2238] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:58.575567 22565 raft_consensus.cc:2267] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:58.613590 22461 tablet_bootstrap.cc:492] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:58.614012 22461 ts_tablet_manager.cc:1397] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.312s user 0.015s sys 0.009s
I20250112 08:56:58.515911 22557 raft_consensus.cc:2238] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:58.626844 22557 raft_consensus.cc:2267] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:58.629666 22556 ts_tablet_manager.cc:1918] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:58.630128 22556 log.cc:1198] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/1b75895a7c9040c1a2ce8c25aeb5c32b
I20250112 08:56:58.630859 22556 ts_tablet_manager.cc:1939] T 1b75895a7c9040c1a2ce8c25aeb5c32b P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:58.528256 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1c634513e14e818601894834c41378. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:58.563715 22149 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.632293 22149 leader_election.cc:304] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:58.447888 22456 raft_consensus.cc:383] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.636580 22456 raft_consensus.cc:738] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.638401 22456 consensus_queue.cc:260] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.644678 22461 raft_consensus.cc:357] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.645351 22461 raft_consensus.cc:383] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.645601 22461 raft_consensus.cc:738] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.654136 22565 ts_tablet_manager.cc:1905] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.654062 22461 consensus_queue.cc:260] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:58.661473 21704 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a72d2062bf2a4c79b5a8be974d1b8cb5 consensus state has no leader
I20250112 08:56:58.664642 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 73) took 1220 ms. Trace:
W20250112 08:56:58.665125 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 154789us
I20250112 08:56:58.665170 22556 rpcz_store.cc:276] 0112 08:56:57.444739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:57.444872 (+ 133us) service_pool.cc:225] Handling call
0112 08:56:58.036327 (+591455us) ts_tablet_manager.cc:1145] Deleting tablet 1b75895a7c9040c1a2ce8c25aeb5c32b
0112 08:56:58.036365 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:58.429041 (+392676us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.629419 (+200378us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:58.664560 (+ 35141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":590902}
I20250112 08:56:58.666216 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 14c7056aaf8844da915acec5f9f92eeb (table default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]) successfully deleted
I20250112 08:56:58.680197 22461 ts_tablet_manager.cc:1428] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.066s user 0.007s sys 0.001s
I20250112 08:56:58.680919 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 1b75895a7c9040c1a2ce8c25aeb5c32b (table default.Table_1eb512e21fe24840b674e404ed647af7 [id=6e3c30a85b234bacb8d316ac8475808b]) successfully deleted
I20250112 08:56:58.681373 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet dabc37f08d1948118b98ba25b689bea4
I20250112 08:56:58.682762 21712 catalog_manager.cc:6596] ReplaceTablet: tablet dabc37f08d1948118b98ba25b689bea4 deleted and replaced by tablet 98ddac23210248e3be1aba6591641b2d
I20250112 08:56:58.683174 21712 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56650 (request call id 128) took 1010 ms. Trace:
I20250112 08:56:58.685701 22556 tablet_replica.cc:331] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:58.689083 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_a2e98237a7f34d58875c6af5ab558a33" } new_table_name: "default.Table_8e00eee0aedf49bcbf1528e033fa3620" modify_external_catalogs: true
I20250112 08:56:58.688337 22461 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:58.695611 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f43c29fa6315404ca33380d0034bf1b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:58.704838 22556 raft_consensus.cc:2238] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 08:56:58.706857 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f43c29fa6315404ca33380d0034bf1b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:58.707249 22556 raft_consensus.cc:2267] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:58.683512 21712 rpcz_store.cc:276] 0112 08:56:57.672553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:57.672658 (+ 105us) service_pool.cc:225] Handling call
0112 08:56:57.684514 (+ 11856us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26576d31 00007fcf242f8d47 00007fcf242f8cd2 00007fcf242f8c01 00007fcf242f8b7c 00007fcf242f7e32 00007fcf2259adfc 00007fcf2259a236
0112 08:56:58.683089 (+998575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:58.493775 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:58.497809 (+ 4034us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:58.497849 (+ 40us) write_op.cc:435] Acquired schema lock
0112 08:56:58.497861 (+ 12us) tablet.cc:659] Decoding operations
0112 08:56:58.498165 (+ 304us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:58.498200 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:58.498223 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:56:58.498416 (+ 193us) tablet.cc:698] Row locks acquired
0112 08:56:58.498428 (+ 12us) write_op.cc:260] PREPARE: finished
0112 08:56:58.498542 (+ 114us) write_op.cc:270] Start()
0112 08:56:58.515451 (+ 16909us) write_op.cc:276] Timestamp: P: 1736672218498462 usec, L: 0
0112 08:56:58.515468 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 08:56:58.516021 (+ 553us) log.cc:844] Serialized 628 byte log entry
0112 08:56:58.664543 (+148522us) op_driver.cc:464] REPLICATION: finished
0112 08:56:58.678362 (+ 13819us) write_op.cc:301] APPLY: starting
0112 08:56:58.678424 (+ 62us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:58.678537 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:58.678648 (+ 111us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:58.679042 (+ 394us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:58.679255 (+ 213us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:56:58.679293 (+ 38us) write_op.cc:312] APPLY: finished
0112 08:56:58.679480 (+ 187us) log.cc:844] Serialized 29 byte log entry
0112 08:56:58.679636 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:58.679708 (+ 72us) write_op.cc:454] Released schema lock
0112 08:56:58.679739 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0112 08:56:58.679974 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:58.679999 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":24807808,"child_traces":[["op",{"apply.queue_time_us":375,"apply.run_cpu_time_us":1584,"apply.run_wall_time_us":1584,"num_ops":2,"prepare.queue_time_us":374435,"prepare.run_cpu_time_us":2200,"prepare.run_wall_time_us":30786,"raft.queue_time_us":11622,"raft.run_cpu_time_us":699,"raft.run_wall_time_us":698,"replication_time_us":28850}]]}
I20250112 08:56:58.715422 22556 ts_tablet_manager.cc:1905] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.725791 22461 tablet_bootstrap.cc:654] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.736227 22666 raft_consensus.cc:491] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.736668 22666 raft_consensus.cc:513] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.738571 22666 leader_election.cc:290] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:58.741101 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:58.667815 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]) successfully deleted
W20250112 08:56:58.741796 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 73989.3us
W20250112 08:56:58.743000 22002 leader_election.cc:343] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.743296 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet dabc37f08d1948118b98ba25b689bea4 with delete_type TABLET_DATA_DELETED (replaced by tablet 98ddac23210248e3be1aba6591641b2d) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:58.745747 22461 tablet_bootstrap.cc:492] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:58.746245 22461 ts_tablet_manager.cc:1397] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.000s
I20250112 08:56:58.756595 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 815d246b7d414aadbb69ac692a5d40c7 (DEFAULT_TABLE table=default.Table_be70d2c098e64ac4ac3f2fef255d0a3d [id=1460b936c99743a3aca81d2d2a377fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:58.757659 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 815d246b7d414aadbb69ac692a5d40c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.770282 22461 raft_consensus.cc:357] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.771176 22461 raft_consensus.cc:383] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.771503 22461 raft_consensus.cc:738] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.639415 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9848b879a2e40aeb840924261e6a100. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:58.661051 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 97396.5us
I20250112 08:56:58.781801 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267ab3e5955b4d3bbeca44baf021b328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:58.661728 22662 raft_consensus.cc:491] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.662549 22698 raft_consensus.cc:2743] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.739683 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:58.783257 22212 raft_consensus.cc:2463] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:58.783215 22662 raft_consensus.cc:513] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.784977 21999 leader_election.cc:304] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:58.780426 22461 consensus_queue.cc:260] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.785984 22666 raft_consensus.cc:2798] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:58.786288 22666 raft_consensus.cc:491] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:58.786401 22662 leader_election.cc:290] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:56:58.786569 22666 raft_consensus.cc:3054] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:58.789245 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:58.790612 22456 ts_tablet_manager.cc:1428] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.347s user 0.008s sys 0.000s
I20250112 08:56:58.792807 22461 ts_tablet_manager.cc:1428] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.046s user 0.008s sys 0.001s
W20250112 08:56:58.792943 22146 leader_election.cc:343] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.801308 22557 ts_tablet_manager.cc:1905] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.837692 22666 raft_consensus.cc:513] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.843396 22666 leader_election.cc:290] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:58.843636 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:58.844409 22212 raft_consensus.cc:3054] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:58.847891 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:56:58.850167 21708 master_service.cc:946] ReplaceTablet: received request to replace tablet 65591a0bb44148c49715b4564dd13524 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:58.855266 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:57 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:58.856878 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:57 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:58.547812 22458 tablet_bootstrap.cc:654] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:58.888393 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:58.890276 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:58.891963 22149 leader_election.cc:343] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:58.892365 22149 leader_election.cc:304] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:58.893039 22662 raft_consensus.cc:2743] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.897182 22456 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:58.900436 22461 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:58.904855 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e85c484e045548ada7a5fbc4545abbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:58.906559 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e85c484e045548ada7a5fbc4545abbd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:58.907142 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet dabc37f08d1948118b98ba25b689bea4 with delete_type TABLET_DATA_DELETED (replaced by tablet 98ddac23210248e3be1aba6591641b2d) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:58.910068 22461 tablet_bootstrap.cc:654] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:58.912201 22002 leader_election.cc:343] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:56:58.915074 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e19979ef29944405b833bd6dad6f3664 consensus state has no leader
I20250112 08:56:58.920799 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7971251621ad4bb69351f4beb52fa190" } modify_external_catalogs: true
I20250112 08:56:58.927985 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7971251621ad4bb69351f4beb52fa190" } modify_external_catalogs: true
I20250112 08:56:58.928308 22183 tablet_service.cc:1514] Processing DeleteTablet for tablet dabc37f08d1948118b98ba25b689bea4 with delete_type TABLET_DATA_DELETED (replaced by tablet 98ddac23210248e3be1aba6591641b2d) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:58.929152 22182 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:57 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:58.936698 22556 ts_tablet_manager.cc:1918] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:56:58.937139 22556 log.cc:1198] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:56:58.937851 22556 ts_tablet_manager.cc:1939] T aa641fe365844c9eaa6ffcbf548d82f7 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:58.938513 22212 raft_consensus.cc:2463] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:58.939777 21999 leader_election.cc:304] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:58.941963 22461 tablet_bootstrap.cc:492] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:58.942427 22461 ts_tablet_manager.cc:1397] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20250112 08:56:58.943153 22556 tablet_replica.cc:331] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:58.944195 22556 raft_consensus.cc:2238] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:58.945040 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]) successfully deleted
I20250112 08:56:58.949366 22556 raft_consensus.cc:2267] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:58.952535 22666 raft_consensus.cc:2798] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:58.954167 22556 ts_tablet_manager.cc:1905] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:58.966243 22458 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:58.944669 22461 raft_consensus.cc:357] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.968781 22461 raft_consensus.cc:383] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.969286 22458 ts_tablet_manager.cc:1397] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.457s user 0.008s sys 0.004s
I20250112 08:56:58.969602 22461 raft_consensus.cc:738] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.970646 22461 consensus_queue.cc:260] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.977090 22461 ts_tablet_manager.cc:1428] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20250112 08:56:58.981189 22458 raft_consensus.cc:357] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.977723 22682 raft_consensus.cc:491] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:58.984591 22682 raft_consensus.cc:513] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:58.986435 22682 leader_election.cc:290] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:56:58.989090 22290 leader_election.cc:343] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:56:58.989614 22458 raft_consensus.cc:383] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:58.990082 22461 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:56:58.989722 22290 leader_election.cc:343] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:56:58.990245 22458 raft_consensus.cc:738] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:58.990657 22290 leader_election.cc:304] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:58.991305 22682 raft_consensus.cc:2743] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:58.993098 22458 consensus_queue.cc:260] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.012710 22666 raft_consensus.cc:695] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:59.013480 22666 consensus_queue.cc:237] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.016615 22458 ts_tablet_manager.cc:1428] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.047s user 0.003s sys 0.003s
I20250112 08:56:59.017330 22461 tablet_bootstrap.cc:654] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.017654 22458 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.021165 22662 raft_consensus.cc:491] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.021733 22662 raft_consensus.cc:513] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.023855 22662 leader_election.cc:290] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:59.025831 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:59.027045 22149 leader_election.cc:343] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.033596 22461 tablet_bootstrap.cc:492] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.034025 22461 ts_tablet_manager.cc:1397] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20250112 08:56:59.035488 22682 raft_consensus.cc:491] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.035943 22682 raft_consensus.cc:513] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.037673 22682 leader_election.cc:290] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:56:59.040915 22290 leader_election.cc:343] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:56:59.042322 22461 raft_consensus.cc:357] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.042963 22461 raft_consensus.cc:383] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.043181 22461 raft_consensus.cc:738] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.043783 22461 consensus_queue.cc:260] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.045740 22461 ts_tablet_manager.cc:1428] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250112 08:56:59.052114 22461 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:59.069196 22461 tablet_bootstrap.cc:654] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.078809 22456 tablet_bootstrap.cc:654] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.084206 22461 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.084906 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0192d66e64d4d39b4bcfd6565dff723" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:59.085502 22064 raft_consensus.cc:2463] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:59.086706 22290 leader_election.cc:304] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:56:59.087527 22682 raft_consensus.cc:2798] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:59.087848 22682 raft_consensus.cc:491] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:59.088294 22682 raft_consensus.cc:3054] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.090601 22458 tablet_bootstrap.cc:654] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.097707 22461 ts_tablet_manager.cc:1397] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.000s
I20250112 08:56:59.101011 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.103111 22146 leader_election.cc:343] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.103595 22146 leader_election.cc:304] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:59.104712 22695 raft_consensus.cc:2743] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:59.105850 22146 consensus_peers.cc:487] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:59.122925 22461 raft_consensus.cc:357] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.123767 22461 raft_consensus.cc:383] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.124830 22461 raft_consensus.cc:738] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.127892 22458 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.128384 22458 ts_tablet_manager.cc:1397] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.004s
I20250112 08:56:59.130594 22458 raft_consensus.cc:357] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.131356 22458 raft_consensus.cc:383] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.131649 22458 raft_consensus.cc:738] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.132274 22458 consensus_queue.cc:260] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.134311 22458 ts_tablet_manager.cc:1428] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250112 08:56:59.126247 22461 consensus_queue.cc:260] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.138077 22461 ts_tablet_manager.cc:1428] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20250112 08:56:59.139003 22458 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.139341 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_d3832ac137d5425db9a0f35eb56cfd68" } new_table_name: "default.Table_4cbe685a04de4b6f8f391e827247cfee" modify_external_catalogs: true
W20250112 08:56:59.141273 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53167910ab0a4accb1d1615d537ba7ae has no consensus state
I20250112 08:56:59.140944 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_83fb15dd26114a99b446cc4d91c8dd14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:59.143543 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83fb15dd26114a99b446cc4d91c8dd14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:59.142129 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0432088806604c248cefd166a4922284" } modify_external_catalogs: true
I20250112 08:56:59.144243 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0432088806604c248cefd166a4922284" } modify_external_catalogs: true
I20250112 08:56:59.146624 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_304a19a1f72e4fbd984a0b9156468aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:59.148150 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_304a19a1f72e4fbd984a0b9156468aa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:59.151357 22682 raft_consensus.cc:513] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.173324 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0192d66e64d4d39b4bcfd6565dff723" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:59.173555 22682 leader_election.cc:290] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.173961 22064 raft_consensus.cc:3054] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.174345 22682 raft_consensus.cc:491] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.174741 22682 raft_consensus.cc:513] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.178663 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.180276 22290 leader_election.cc:343] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:56:59.179811 22682 leader_election.cc:290] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.181790 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8509fdc7d6b467ab8d445f8a15fa014" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:59.182308 22461 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:59.182610 22213 raft_consensus.cc:2388] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 76718d1596134c83900541c6d398d59e in this term.
I20250112 08:56:59.184926 22290 leader_election.cc:304] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:59.191579 22458 tablet_bootstrap.cc:654] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.193291 22716 raft_consensus.cc:3054] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.194782 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet c08341b5ce6040d5be30f324d0373e38 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:59.197162 22461 tablet_bootstrap.cc:654] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.210219 22458 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.210611 22458 ts_tablet_manager.cc:1397] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.001s
I20250112 08:56:59.214903 22461 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.215274 22461 ts_tablet_manager.cc:1397] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.001s
I20250112 08:56:59.217931 22461 raft_consensus.cc:357] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.218585 22461 raft_consensus.cc:383] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.218832 22461 raft_consensus.cc:738] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.221643 22458 raft_consensus.cc:357] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.222287 22458 raft_consensus.cc:383] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.222543 22458 raft_consensus.cc:738] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.224184 22456 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:59.225322 22458 consensus_queue.cc:260] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.227826 22456 ts_tablet_manager.cc:1397] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.331s user 0.011s sys 0.000s
I20250112 08:56:59.236729 22716 raft_consensus.cc:2743] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:59.240502 22064 raft_consensus.cc:2463] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:59.241732 22290 leader_election.cc:304] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:56:59.242882 22716 raft_consensus.cc:2798] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:59.243288 22716 raft_consensus.cc:695] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:56:59.243486 22458 ts_tablet_manager.cc:1428] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250112 08:56:59.244069 22716 consensus_queue.cc:237] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.246546 22461 consensus_queue.cc:260] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.243331 22456 raft_consensus.cc:357] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.247320 22456 raft_consensus.cc:383] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.247577 22456 raft_consensus.cc:738] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.249831 22710 raft_consensus.cc:491] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.250331 22710 raft_consensus.cc:513] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.251514 22461 ts_tablet_manager.cc:1428] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20250112 08:56:59.253779 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.255718 22290 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.257534 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:59.258116 22213 raft_consensus.cc:2463] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:59.259843 22290 leader_election.cc:304] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:56:59.260666 22682 raft_consensus.cc:2798] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:59.260982 22682 raft_consensus.cc:491] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:59.261261 22682 raft_consensus.cc:3054] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.261179 22458 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.262521 22456 consensus_queue.cc:260] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.264822 22666 raft_consensus.cc:491] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.265246 22666 raft_consensus.cc:513] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.266417 22461 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:59.267117 22666 leader_election.cc:290] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.269608 22565 ts_tablet_manager.cc:1918] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:56:59.270018 22565 log.cc:1198] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/6ffbab945abd4fa58259a65e4fadd622
W20250112 08:56:59.270280 21999 leader_election.cc:343] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:56:59.271782 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c683c8c64a441e8b5be52b077fdcb08" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:59.272393 22356 raft_consensus.cc:2463] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:56:59.273381 22461 tablet_bootstrap.cc:654] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.273530 22002 leader_election.cc:304] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:56:59.276531 22666 raft_consensus.cc:2798] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:59.276855 22666 raft_consensus.cc:491] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:59.277145 22666 raft_consensus.cc:3054] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.280660 22458 tablet_bootstrap.cc:654] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.283106 22706 raft_consensus.cc:491] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.283545 22706 raft_consensus.cc:513] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.284790 22710 leader_election.cc:290] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.270934 22565 ts_tablet_manager.cc:1939] T 6ffbab945abd4fa58259a65e4fadd622 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:56:59.285527 22706 leader_election.cc:290] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:56:59.287730 21999 leader_election.cc:343] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a090d258647b4dd7bd91cf64011735dd
I20250112 08:56:59.289227 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090d258647b4dd7bd91cf64011735dd" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:56:59.292188 22002 leader_election.cc:343] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.292549 22002 leader_election.cc:304] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:56:59.293385 21703 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.253:41411, user_credentials={real_user=slave}} blocked reactor thread for 54915.9us
I20250112 08:56:59.293625 22706 raft_consensus.cc:2743] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:59.298043 21708 master_service.cc:946] ReplaceTablet: received request to replace tablet dabc37f08d1948118b98ba25b689bea4 from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:59.298077 22456 ts_tablet_manager.cc:1428] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.070s user 0.005s sys 0.000s
I20250112 08:56:59.298367 22461 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.298741 22461 ts_tablet_manager.cc:1397] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.005s
I20250112 08:56:59.298951 22456 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:59.300825 22461 raft_consensus.cc:357] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.301440 22461 raft_consensus.cc:383] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.301700 22461 raft_consensus.cc:738] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.302193 22461 consensus_queue.cc:260] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.305752 22458 tablet_bootstrap.cc:492] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.306126 22458 ts_tablet_manager.cc:1397] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.003s
I20250112 08:56:59.308336 22458 raft_consensus.cc:357] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.308938 22458 raft_consensus.cc:383] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.309166 22458 raft_consensus.cc:738] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.314628 22458 consensus_queue.cc:260] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.317081 22461 ts_tablet_manager.cc:1428] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20250112 08:56:59.317926 22461 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:59.322023 22458 ts_tablet_manager.cc:1428] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250112 08:56:59.322906 22458 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.326427 21709 catalog_manager.cc:5526] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:56:59.328824 21714 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_7971251621ad4bb69351f4beb52fa190 [id=f69f93bdf75d4f1fb836d8961cbc68c4] alter complete (version 2)
W20250112 08:56:59.331113 21714 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 31) took 1291 ms (client timeout 1000 ms). Trace:
I20250112 08:56:59.335289 22458 tablet_bootstrap.cc:654] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:59.331434 21714 rpcz_store.cc:269] 0112 08:56:58.039838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.040020 (+ 182us) service_pool.cc:225] Handling call
0112 08:56:58.093742 (+ 53722us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26572dee 00007fcf242d0fc4 00007fcf242d0f52 00007fcf242d0e81 00007fcf242d0dfc 00007fcf242d00b2 00007fcf2259adfc 00007fcf2259a236
0112 08:56:59.331029 (+1237287us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:58.718433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:58.718853 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:58.718894 (+ 41us) write_op.cc:435] Acquired schema lock
0112 08:56:58.718911 (+ 17us) tablet.cc:659] Decoding operations
0112 08:56:58.719335 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:58.719369 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:58.719391 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:56:58.719538 (+ 147us) tablet.cc:698] Row locks acquired
0112 08:56:58.719552 (+ 14us) write_op.cc:260] PREPARE: finished
0112 08:56:58.719651 (+ 99us) write_op.cc:270] Start()
0112 08:56:58.719740 (+ 89us) write_op.cc:276] Timestamp: P: 1736672218719563 usec, L: 0
0112 08:56:58.719755 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 08:56:58.720292 (+ 537us) log.cc:844] Serialized 356 byte log entry
0112 08:56:58.726231 (+ 5939us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b58000905b0. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf24747a94 00007fcf2474c9d1 00007fcf2474c959 00007fcf2474c8e9 00007fcf2474c8b1 00007fcf2474ba7d 000000000059ea04 000000000059e839 00007fcf1d885156 00007fcf1d8885b1
0112 08:56:58.844667 (+118436us) op_driver.cc:464] REPLICATION: finished
0112 08:56:58.846787 (+ 2120us) write_op.cc:301] APPLY: starting
0112 08:56:58.846837 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:58.846914 (+ 77us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:58.846933 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:58.847101 (+ 168us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:58.847217 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:56:58.847255 (+ 38us) write_op.cc:312] APPLY: finished
0112 08:56:58.847449 (+ 194us) log.cc:844] Serialized 23 byte log entry
0112 08:56:58.849000 (+ 1551us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:58.849090 (+ 90us) write_op.cc:454] Released schema lock
0112 08:56:58.849135 (+ 45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":116842880,"child_traces":[["op",{"apply.queue_time_us":1973,"apply.run_cpu_time_us":1620,"apply.run_wall_time_us":2842,"num_ops":1,"prepare.queue_time_us":565099,"prepare.run_cpu_time_us":2049,"prepare.run_wall_time_us":3701,"raft.queue_time_us":77,"raft.run_cpu_time_us":693,"raft.run_wall_time_us":4340,"replication_time_us":124826,"spinlock_wait_cycles":8644992,"thread_start_us":56543,"threads_started":1}]]}
I20250112 08:56:59.337342 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 02538a000a0f4fa5a40f1ab5cd862fe1 (DEFAULT_TABLE table=default.Table_b8ef9dfb2cb1497bb83b09e8bff95c68 [id=325d148802b34ab8b04df0516a2740d9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:56:59.338138 21711 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 32) took 829 ms (client timeout 1000 ms). Trace:
I20250112 08:56:59.338366 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02538a000a0f4fa5a40f1ab5cd862fe1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:59.338477 21711 rpcz_store.cc:269] 0112 08:56:58.508488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.508642 (+ 154us) service_pool.cc:225] Handling call
0112 08:56:59.337959 (+829317us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
I20250112 08:56:59.338564 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 02538a000a0f4fa5a40f1ab5cd862fe1 (DEFAULT_TABLE table=default.Table_b8ef9dfb2cb1497bb83b09e8bff95c68 [id=325d148802b34ab8b04df0516a2740d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.339478 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02538a000a0f4fa5a40f1ab5cd862fe1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:56:59.343724 21715 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 31) took 834 ms (client timeout 1000 ms). Trace:
W20250112 08:56:59.344166 21715 rpcz_store.cc:269] 0112 08:56:58.508814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.508993 (+ 179us) service_pool.cc:225] Handling call
0112 08:56:59.343566 (+834573us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4164,"reported_tablets":5}
I20250112 08:56:59.345062 22335 tablet_service.cc:1467] Processing CreateTablet for tablet f61aedea18d944179e788c08488c0a5e (DEFAULT_TABLE table=default.Table_62b29cf7e29a49a587abdbe532a560c4 [id=03f81997bd3a467a812dfcdee6f272cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.346124 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61aedea18d944179e788c08488c0a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.345343 22682 raft_consensus.cc:513] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.347877 22456 tablet_bootstrap.cc:654] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.350384 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:59.352571 22040 tablet_service.cc:1467] Processing CreateTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.353086 22331 tablet_service.cc:1467] Processing CreateTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.353616 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44cad3bd5a84e75a41733d8d2f345a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.354099 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44cad3bd5a84e75a41733d8d2f345a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.352931 22047 tablet_service.cc:1467] Processing CreateTablet for tablet f61aedea18d944179e788c08488c0a5e (DEFAULT_TABLE table=default.Table_62b29cf7e29a49a587abdbe532a560c4 [id=03f81997bd3a467a812dfcdee6f272cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.356673 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61aedea18d944179e788c08488c0a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.357057 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (DEFAULT_TABLE table=default.Table_51785214aefc4cb0abcc3983ff977c40 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.357939 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.358707 22456 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:59.359100 22456 ts_tablet_manager.cc:1397] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.010s
I20250112 08:56:59.360814 22191 tablet_service.cc:1467] Processing CreateTablet for tablet f61aedea18d944179e788c08488c0a5e (DEFAULT_TABLE table=default.Table_62b29cf7e29a49a587abdbe532a560c4 [id=03f81997bd3a467a812dfcdee6f272cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.361299 22456 raft_consensus.cc:357] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.361963 22456 raft_consensus.cc:383] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.362200 22456 raft_consensus.cc:738] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.362166 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 02538a000a0f4fa5a40f1ab5cd862fe1 (DEFAULT_TABLE table=default.Table_b8ef9dfb2cb1497bb83b09e8bff95c68 [id=325d148802b34ab8b04df0516a2740d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.362960 22456 consensus_queue.cc:260] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.363615 22184 tablet_service.cc:1467] Processing CreateTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.363749 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (DEFAULT_TABLE table=default.Table_51785214aefc4cb0abcc3983ff977c40 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.364773 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.365978 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:59.367163 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:56:59.368654 22329 tablet_service.cc:1467] Processing CreateTablet for tablet 53167910ab0a4accb1d1615d537ba7ae (DEFAULT_TABLE table=default.Table_8e00eee0aedf49bcbf1528e033fa3620 [id=1b9b675047ab4ad3a5b0d3028a578303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.369729 22329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53167910ab0a4accb1d1615d537ba7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.373492 22456 ts_tablet_manager.cc:1428] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250112 08:56:59.374231 22682 leader_election.cc:290] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.374888 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:59.377415 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:56:59.377941 22213 raft_consensus.cc:3054] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.382136 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]) successfully deleted
I20250112 08:56:59.383472 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:59.384985 22456 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:59.388559 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (DEFAULT_TABLE table=default.Table_51785214aefc4cb0abcc3983ff977c40 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.390506 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 53167910ab0a4accb1d1615d537ba7ae (DEFAULT_TABLE table=default.Table_8e00eee0aedf49bcbf1528e033fa3620 [id=1b9b675047ab4ad3a5b0d3028a578303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.391191 22456 tablet_bootstrap.cc:654] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.391525 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53167910ab0a4accb1d1615d537ba7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.396271 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_3bb2b43e854340fba2b4c4ad8a3da14a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:59.400019 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61aedea18d944179e788c08488c0a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.401259 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.401597 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 53167910ab0a4accb1d1615d537ba7ae (DEFAULT_TABLE table=default.Table_8e00eee0aedf49bcbf1528e033fa3620 [id=1b9b675047ab4ad3a5b0d3028a578303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.402930 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 1514074a19694aa0a42b968a29904c30 (DEFAULT_TABLE table=default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.403298 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44cad3bd5a84e75a41733d8d2f345a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.403329 22042 tablet_service.cc:1467] Processing CreateTablet for tablet 1514074a19694aa0a42b968a29904c30 (DEFAULT_TABLE table=default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.404538 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1514074a19694aa0a42b968a29904c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.404682 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02538a000a0f4fa5a40f1ab5cd862fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.404384 22046 tablet_service.cc:1467] Processing CreateTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 (DEFAULT_TABLE table=default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.405414 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b38deeff1964b09938a12cfd4992ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.409279 22187 tablet_service.cc:1467] Processing CreateTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 (DEFAULT_TABLE table=default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.410303 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b38deeff1964b09938a12cfd4992ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.419739 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53167910ab0a4accb1d1615d537ba7ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.420935 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1514074a19694aa0a42b968a29904c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.434798 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 1514074a19694aa0a42b968a29904c30 (DEFAULT_TABLE table=default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.435729 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1514074a19694aa0a42b968a29904c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.436568 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_51785214aefc4cb0abcc3983ff977c40" } new_table_name: "default.Table_ec21905b8bd64621b9687023c5d3024e" modify_external_catalogs: true
I20250112 08:56:59.446350 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 (DEFAULT_TABLE table=default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:56:59.447391 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b38deeff1964b09938a12cfd4992ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:56:59.449386 22557 ts_tablet_manager.cc:1918] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:56:59.449812 22557 log.cc:1198] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/7451e83f3b554936a79c6724ebe76cbb
I20250112 08:56:59.450589 22557 ts_tablet_manager.cc:1939] T 7451e83f3b554936a79c6724ebe76cbb P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:56:59.454037 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet dabc37f08d1948118b98ba25b689bea4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dabc37f08d1948118b98ba25b689bea4 already in progress: creating tablet
W20250112 08:56:59.464144 22290 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.468971 22213 raft_consensus.cc:2463] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:59.470145 22290 leader_election.cc:304] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:56:59.476142 22682 raft_consensus.cc:2798] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:59.477710 22565 tablet_replica.cc:331] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:56:59.480612 22682 raft_consensus.cc:695] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:56:59.481928 22556 ts_tablet_manager.cc:1918] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:56:59.482247 22556 log.cc:1198] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/6ffbab945abd4fa58259a65e4fadd622
I20250112 08:56:59.482947 22556 ts_tablet_manager.cc:1939] T 6ffbab945abd4fa58259a65e4fadd622 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:56:59.484058 22682 consensus_queue.cc:237] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.485354 22565 raft_consensus.cc:2238] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:59.476756 22666 raft_consensus.cc:513] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:59.490092 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bb2b43e854340fba2b4c4ad8a3da14a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:59.490617 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c683c8c64a441e8b5be52b077fdcb08" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:56:59.491137 22356 raft_consensus.cc:3054] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.492949 22565 raft_consensus.cc:2267] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:59.523639 22456 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:59.524170 22456 ts_tablet_manager.cc:1397] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.139s user 0.020s sys 0.044s
I20250112 08:56:59.526535 22456 raft_consensus.cc:357] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.527308 22456 raft_consensus.cc:383] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.527608 22456 raft_consensus.cc:738] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.474047 22461 tablet_bootstrap.cc:654] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.528340 22456 consensus_queue.cc:260] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.536756 22458 tablet_bootstrap.cc:492] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.536406 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_7199b4dc8d794b8a813e20147ba3f6d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:56:59.537153 22458 ts_tablet_manager.cc:1397] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.214s user 0.018s sys 0.010s
W20250112 08:56:59.537919 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7199b4dc8d794b8a813e20147ba3f6d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:59.539249 22458 raft_consensus.cc:357] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.539885 22458 raft_consensus.cc:383] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.553043 22456 ts_tablet_manager.cc:1428] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20250112 08:56:59.552639 22458 raft_consensus.cc:738] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.554373 22458 consensus_queue.cc:260] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.557754 22356 raft_consensus.cc:2463] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:56:59.558797 22002 leader_election.cc:304] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:59.563325 22706 raft_consensus.cc:2798] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:59.563769 22706 raft_consensus.cc:695] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:56:59.564039 22710 raft_consensus.cc:491] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.564504 22706 consensus_queue.cc:237] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.564783 22710 raft_consensus.cc:513] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.567397 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08341b5ce6040d5be30f324d0373e38" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.568703 22290 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.574836 22456 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:59.577235 22695 raft_consensus.cc:491] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.577697 22695 raft_consensus.cc:513] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.582350 22695 leader_election.cc:290] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:59.583355 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.584583 22146 leader_election.cc:343] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.585453 22458 ts_tablet_manager.cc:1428] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.048s user 0.001s sys 0.005s
W20250112 08:56:59.586618 22290 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38
I20250112 08:56:59.587059 22290 leader_election.cc:304] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:56:59.588394 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:59.589301 22716 raft_consensus.cc:2743] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:56:59.590324 22149 leader_election.cc:343] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.590770 22149 leader_election.cc:304] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:59.592227 22695 raft_consensus.cc:2743] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:59.593127 22710 leader_election.cc:290] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.596480 22458 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.598775 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 83) took 2027 ms. Trace:
I20250112 08:56:59.599368 22557 rpcz_store.cc:276] 0112 08:56:57.571597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:57.589265 (+ 17668us) service_pool.cc:225] Handling call
0112 08:56:58.515067 (+925802us) ts_tablet_manager.cc:1145] Deleting tablet 7451e83f3b554936a79c6724ebe76cbb
0112 08:56:58.515121 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.156534 (+641413us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:59.449106 (+292572us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:59.598719 (+149613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":914456}
I20250112 08:56:59.600833 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 7451e83f3b554936a79c6724ebe76cbb (table default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]) successfully deleted
I20250112 08:56:59.605032 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 82) took 2034 ms. Trace:
I20250112 08:56:59.608350 22461 tablet_bootstrap.cc:492] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.608819 22461 ts_tablet_manager.cc:1397] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.291s user 0.007s sys 0.006s
I20250112 08:56:59.611101 22461 raft_consensus.cc:357] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.611817 22461 raft_consensus.cc:383] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.612138 22461 raft_consensus.cc:738] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.612844 22461 consensus_queue.cc:260] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.617326 22458 tablet_bootstrap.cc:654] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:56:59.620222 21999 leader_election.cc:343] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:56:59.620802 22666 leader_election.cc:290] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.623296 22565 ts_tablet_manager.cc:1905] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:59.654618 22458 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.655035 22458 ts_tablet_manager.cc:1397] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.009s
I20250112 08:56:59.665072 22458 raft_consensus.cc:357] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.665722 22458 raft_consensus.cc:383] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.665963 22458 raft_consensus.cc:738] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.668766 22458 consensus_queue.cc:260] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.674253 22458 ts_tablet_manager.cc:1428] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250112 08:56:59.675065 22458 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:56:59.689175 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9848b879a2e40aeb840924261e6a100 consensus state has no leader
I20250112 08:56:59.690016 21716 master_service.cc:946] ReplaceTablet: received request to replace tablet 06717cb567a54ee6a268c3be5d8786ec from {username='slave'} at 127.0.0.1:56650
I20250112 08:56:59.690732 22458 tablet_bootstrap.cc:654] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.707957 21713 master_service.cc:946] ReplaceTablet: received request to replace tablet 14c7056aaf8844da915acec5f9f92eeb from {username='slave'} at 127.0.0.1:56650
W20250112 08:56:59.709363 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 574999cd047643d3840a16f2baf6e9bb consensus state has no leader
I20250112 08:56:59.622661 22456 tablet_bootstrap.cc:654] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.717768 22557 rpcz_store.cc:276] 0112 08:56:57.571090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:57.590303 (+ 19213us) service_pool.cc:225] Handling call
0112 08:56:59.604791 (+2014488us) ts_tablet_manager.cc:1145] Deleting tablet a090d258647b4dd7bd91cf64011735dd
0112 08:56:59.604828 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.604984 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003915}
W20250112 08:56:59.719283 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet a090d258647b4dd7bd91cf64011735dd because the tablet was not found. No further retry: Not found: Tablet not found: a090d258647b4dd7bd91cf64011735dd
I20250112 08:56:59.723377 22557 tablet_replica.cc:331] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:56:59.724330 22557 raft_consensus.cc:2238] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:59.724920 22557 raft_consensus.cc:2267] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:59.726580 22557 ts_tablet_manager.cc:1905] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:59.746732 22458 tablet_bootstrap.cc:492] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.747114 22458 ts_tablet_manager.cc:1397] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.013s
I20250112 08:56:59.749188 22458 raft_consensus.cc:357] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.749809 22458 raft_consensus.cc:383] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.750062 22458 raft_consensus.cc:738] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.759290 22458 consensus_queue.cc:260] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.769284 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 35f60d1f4bed4721b73deb9ba787417c
I20250112 08:56:59.770862 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:56:59.771637 22186 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:56:59.763612 22458 ts_tablet_manager.cc:1428] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250112 08:56:59.774144 22458 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.780997 22458 tablet_bootstrap.cc:654] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.781692 22695 raft_consensus.cc:491] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.782146 22695 raft_consensus.cc:513] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:59.788558 22149 consensus_peers.cc:487] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ffbab945abd4fa58259a65e4fadd622. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:56:59.789074 22695 leader_election.cc:290] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:56:59.789599 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:56:59.791431 22146 leader_election.cc:343] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.801962 22461 ts_tablet_manager.cc:1428] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.193s user 0.005s sys 0.000s
I20250112 08:56:59.793001 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:56:59.785601 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:56:59.805306 22149 leader_election.cc:343] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:56:59.805678 22149 leader_election.cc:304] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:56:59.806670 22695 raft_consensus.cc:2743] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:56:59.807583 22461 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:56:59.630059 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 87) took 1217 ms. Trace:
I20250112 08:56:59.824200 22556 rpcz_store.cc:276] 0112 08:56:58.412677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.412997 (+ 320us) service_pool.cc:225] Handling call
0112 08:56:58.942941 (+529944us) ts_tablet_manager.cc:1145] Deleting tablet 6ffbab945abd4fa58259a65e4fadd622
0112 08:56:58.943001 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.205744 (+262743us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:59.481717 (+275973us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:59.630000 (+148283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":528832}
I20250112 08:56:59.826035 22556 tablet_replica.cc:331] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:56:59.826185 22461 tablet_bootstrap.cc:654] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.826793 22556 raft_consensus.cc:2238] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:56:59.827633 22556 raft_consensus.cc:2267] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:56:59.833637 22456 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:56:59.834038 22456 ts_tablet_manager.cc:1397] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.259s user 0.011s sys 0.006s
I20250112 08:56:59.836326 22456 raft_consensus.cc:357] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.836961 22456 raft_consensus.cc:383] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.837188 22456 raft_consensus.cc:738] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.840162 22456 consensus_queue.cc:260] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:56:59.843240 22290 raft_consensus.cc:685] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: failed to submit failure detected task: Service unavailable: Thread pool token was shut down
I20250112 08:56:59.846105 22556 ts_tablet_manager.cc:1905] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:56:59.858191 22456 ts_tablet_manager.cc:1428] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250112 08:56:59.861583 22458 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.862107 22458 ts_tablet_manager.cc:1397] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.019s
I20250112 08:56:59.865072 22458 raft_consensus.cc:357] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.865760 22458 raft_consensus.cc:383] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.866010 22458 raft_consensus.cc:738] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.866596 22458 consensus_queue.cc:260] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.872174 22458 ts_tablet_manager.cc:1428] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20250112 08:56:59.873013 22458 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.878927 22458 tablet_bootstrap.cc:654] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.883227 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_38c3f75d2c224c5fb9b7d1223a62721d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:56:59.885099 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38c3f75d2c224c5fb9b7d1223a62721d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:56:59.893539 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cc848572e4f74f148ed5aa8bb6f16da3" } new_table_name: "default.Table_bbbd78fbf43744eb9ce8bcc320e9755a" modify_external_catalogs: true
I20250112 08:56:59.904515 22456 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:56:59.908565 22458 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.909186 22458 ts_tablet_manager.cc:1397] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.008s
I20250112 08:56:59.911655 22458 raft_consensus.cc:357] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.912500 22458 raft_consensus.cc:383] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.921123 22682 raft_consensus.cc:491] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:56:59.921655 22458 raft_consensus.cc:738] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.921600 22682 raft_consensus.cc:513] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.922732 22458 consensus_queue.cc:260] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.923882 22682 leader_election.cc:290] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.925318 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:56:59.925567 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:56:59.925876 22064 raft_consensus.cc:2463] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:59.926122 22213 raft_consensus.cc:2463] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:56:59.927294 22290 leader_election.cc:304] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:59.928678 22682 raft_consensus.cc:2798] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:56:59.929008 22682 raft_consensus.cc:491] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:56:59.929318 22682 raft_consensus.cc:3054] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.931720 22458 ts_tablet_manager.cc:1428] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.022s user 0.003s sys 0.003s
I20250112 08:56:59.935263 22458 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.940189 22682 raft_consensus.cc:513] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.944473 22458 tablet_bootstrap.cc:654] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.945588 22461 tablet_bootstrap.cc:492] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:56:59.946056 22461 ts_tablet_manager.cc:1397] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.142s user 0.001s sys 0.011s
I20250112 08:56:59.948575 22461 raft_consensus.cc:357] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.949258 22461 raft_consensus.cc:383] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.949510 22461 raft_consensus.cc:738] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.950117 22461 consensus_queue.cc:260] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.957825 22458 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:56:59.958264 22458 ts_tablet_manager.cc:1397] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20250112 08:56:59.960736 22458 raft_consensus.cc:357] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.961479 22458 raft_consensus.cc:383] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:56:59.961719 22458 raft_consensus.cc:738] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:56:59.964666 22458 consensus_queue.cc:260] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.971186 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:56:59.971745 22064 raft_consensus.cc:3054] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:56:59.973889 22458 ts_tablet_manager.cc:1428] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250112 08:56:59.974821 22458 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:56:59.981191 22064 raft_consensus.cc:2463] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:56:59.984591 22290 leader_election.cc:304] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:56:59.985606 22710 raft_consensus.cc:2798] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:56:59.986124 22710 raft_consensus.cc:695] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:56:59.986938 22710 consensus_queue.cc:237] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:56:59.988957 22458 tablet_bootstrap.cc:654] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:56:59.992311 22461 ts_tablet_manager.cc:1428] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250112 08:56:59.993283 22461 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:00.001152 22456 tablet_bootstrap.cc:654] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.002167 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69651ab160db41b3a187542ac1cd2839" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:00.002735 22213 raft_consensus.cc:3054] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.010265 22666 consensus_queue.cc:1035] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:00.019236 22213 raft_consensus.cc:2463] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:00.038054 22456 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:00.038494 22456 ts_tablet_manager.cc:1397] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.134s user 0.007s sys 0.002s
I20250112 08:57:00.040660 22456 raft_consensus.cc:357] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.044775 22682 leader_election.cc:290] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:56:59.999836 22461 tablet_bootstrap.cc:654] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.053941 22458 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.054509 22458 ts_tablet_manager.cc:1397] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.080s user 0.015s sys 0.014s
I20250112 08:57:00.056723 22458 raft_consensus.cc:357] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.041263 22456 raft_consensus.cc:383] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.057601 22456 raft_consensus.cc:738] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.057768 22458 raft_consensus.cc:383] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.058043 22458 raft_consensus.cc:738] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.058351 22456 consensus_queue.cc:260] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.058575 22458 consensus_queue.cc:260] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.060556 22458 ts_tablet_manager.cc:1428] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250112 08:57:00.067400 22456 ts_tablet_manager.cc:1428] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20250112 08:57:00.076756 22456 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:00.079046 22458 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.083972 22461 tablet_bootstrap.cc:492] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:00.084373 22461 ts_tablet_manager.cc:1397] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.000s
W20250112 08:57:00.085489 21703 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.253:41411, user_credentials={real_user=slave}} blocked reactor thread for 261803us
I20250112 08:57:00.086753 22461 raft_consensus.cc:357] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.087484 22461 raft_consensus.cc:383] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.087755 22461 raft_consensus.cc:738] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.087761 22565 ts_tablet_manager.cc:1918] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:00.088243 22565 log.cc:1198] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:00.093075 22461 consensus_queue.cc:260] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.092933 22458 tablet_bootstrap.cc:654] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.099545 22456 tablet_bootstrap.cc:654] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:00.100765 22290 consensus_peers.cc:487] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0192d66e64d4d39b4bcfd6565dff723. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:00.102962 22461 ts_tablet_manager.cc:1428] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 08:57:00.113987 22458 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.113849 22710 consensus_queue.cc:1035] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:00.114481 22458 ts_tablet_manager.cc:1397] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20250112 08:57:00.114899 22461 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:00.116597 22666 raft_consensus.cc:491] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.117177 22666 raft_consensus.cc:513] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.119359 22666 leader_election.cc:290] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.121191 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08341b5ce6040d5be30f324d0373e38" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:00.121809 22356 raft_consensus.cc:2463] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:00.122112 22458 raft_consensus.cc:357] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.122963 22458 raft_consensus.cc:383] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.123273 22458 raft_consensus.cc:738] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.123168 22002 leader_election.cc:304] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:00.124199 22666 raft_consensus.cc:2798] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.124145 22458 consensus_queue.cc:260] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.124635 22666 raft_consensus.cc:491] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.125000 22666 raft_consensus.cc:3054] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.126112 22458 ts_tablet_manager.cc:1428] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20250112 08:57:00.126888 22458 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.134238 22710 raft_consensus.cc:491] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.134647 22710 raft_consensus.cc:513] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.136807 22710 leader_election.cc:290] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.140190 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7451e83f3b554936a79c6724ebe76cbb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:00.136915 22666 raft_consensus.cc:513] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.141588 22064 raft_consensus.cc:2463] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:00.143270 22461 tablet_bootstrap.cc:654] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:00.143189 22290 leader_election.cc:343] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 7451e83f3b554936a79c6724ebe76cbb
I20250112 08:57:00.144367 22290 leader_election.cc:304] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:00.145123 22710 raft_consensus.cc:2798] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.145476 22710 raft_consensus.cc:491] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.145874 22710 raft_consensus.cc:3054] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:00.146180 21999 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38
I20250112 08:57:00.153489 22666 leader_election.cc:290] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:00.154505 21999 leader_election.cc:343] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38
I20250112 08:57:00.172351 22458 tablet_bootstrap.cc:654] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.175225 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08341b5ce6040d5be30f324d0373e38" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:00.175944 22356 raft_consensus.cc:3054] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.184540 22356 raft_consensus.cc:2463] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:00.188297 22002 leader_election.cc:304] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:00.189993 22666 raft_consensus.cc:2798] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:00.199816 22458 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.200273 22458 ts_tablet_manager.cc:1397] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.006s
I20250112 08:57:00.202675 22458 raft_consensus.cc:357] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.203436 22458 raft_consensus.cc:383] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.212029 22666 raft_consensus.cc:695] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:00.213145 22666 consensus_queue.cc:237] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.203716 22458 raft_consensus.cc:738] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.217270 22557 ts_tablet_manager.cc:1918] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:00.217036 22458 consensus_queue.cc:260] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.142623 22565 ts_tablet_manager.cc:1939] T 7a679595edde4a1f9b7f0916b2d760f2 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:00.142388 22708 consensus_queue.cc:1035] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250112 08:56:58.140393 22633 debug-util.cc:398] Leaking SignalData structure 0x7b0800006bc0 after lost signal to thread 22572
W20250112 08:57:00.221014 22572 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 3.466s user 0.381s sys 0.571s
W20250112 08:57:00.226755 22572 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 3.472s user 0.381s sys 0.571s
I20250112 08:57:00.232154 22557 log.cc:1198] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:57:00.232901 22557 ts_tablet_manager.cc:1939] T aa641fe365844c9eaa6ffcbf548d82f7 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:56:56.761123 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:00.234867 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 86) took 1946 ms. Trace:
I20250112 08:57:00.235154 22557 rpcz_store.cc:276] 0112 08:56:58.288421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.288597 (+ 176us) service_pool.cc:225] Handling call
0112 08:56:59.723187 (+1434590us) ts_tablet_manager.cc:1145] Deleting tablet aa641fe365844c9eaa6ffcbf548d82f7
0112 08:56:59.723241 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.959144 (+235903us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.217052 (+257908us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.234812 (+ 17760us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433912}
I20250112 08:57:00.235561 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 87) took 1380 ms. Trace:
I20250112 08:57:00.241214 22565 rpcz_store.cc:276] 0112 08:56:58.854938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.855084 (+ 146us) service_pool.cc:225] Handling call
0112 08:56:59.477552 (+622468us) ts_tablet_manager.cc:1145] Deleting tablet 7a679595edde4a1f9b7f0916b2d760f2
0112 08:56:59.477595 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.789726 (+312131us) tablet_metadata.cc:641] Metadata flushed
0112 08:56:59.998448 (+208722us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.235505 (+237057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":611443}
W20250112 08:57:00.242457 22763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:00.242637 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 consensus state has no leader
I20250112 08:57:00.245045 21715 master_service.cc:946] ReplaceTablet: received request to replace tablet 010ce6f55c444fe0852ad72cc7383cc1 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:00.247159 22572 server_base.cc:1034] running on GCE node
I20250112 08:57:00.248687 22572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:00.246052 22458 ts_tablet_manager.cc:1428] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.045s user 0.011s sys 0.000s
I20250112 08:57:00.252630 22458 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.257932 22458 tablet_bootstrap.cc:654] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.265692 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25]) successfully deleted
W20250112 08:57:00.267092 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet aa641fe365844c9eaa6ffcbf548d82f7 because the tablet was not found. No further retry: Not found: Tablet not found: aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:57:00.270125 22456 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:00.274528 22456 ts_tablet_manager.cc:1397] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.206s user 0.012s sys 0.011s
I20250112 08:57:00.277457 22456 raft_consensus.cc:357] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.276999 22710 raft_consensus.cc:513] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.279357 22710 leader_election.cc:290] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.284147 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7451e83f3b554936a79c6724ebe76cbb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:00.284720 22068 raft_consensus.cc:3054] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.291072 22068 raft_consensus.cc:2463] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:00.293093 22290 leader_election.cc:304] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:00.296483 22716 raft_consensus.cc:2798] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:00.296905 22716 raft_consensus.cc:695] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:00.298660 22556 ts_tablet_manager.cc:1918] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:00.297642 22716 consensus_queue.cc:237] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:00.307678 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 35f60d1f4bed4721b73deb9ba787417c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f60d1f4bed4721b73deb9ba787417c already in progress: creating tablet
I20250112 08:57:00.314671 22572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250112 08:57:00.315079 22290 leader_election.cc:343] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 7451e83f3b554936a79c6724ebe76cbb
I20250112 08:57:00.310425 22736 raft_consensus.cc:491] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.317097 22456 raft_consensus.cc:383] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.317297 22458 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.317385 22456 raft_consensus.cc:738] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.317682 22458 ts_tablet_manager.cc:1397] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.004s
I20250112 08:57:00.317231 22736 raft_consensus.cc:513] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.320133 22456 consensus_queue.cc:260] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.321933 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.323133 22149 leader_election.cc:343] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.323290 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:00.324539 22146 leader_election.cc:343] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.324915 22146 leader_election.cc:304] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.325584 22695 raft_consensus.cc:2743] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.320518 22736 leader_election.cc:290] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:00.327529 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet aa641fe365844c9eaa6ffcbf548d82f7 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]) successfully deleted
I20250112 08:57:00.327776 22557 tablet_replica.cc:331] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:00.327813 22458 raft_consensus.cc:357] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.328454 22557 raft_consensus.cc:2238] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:00.328720 22458 raft_consensus.cc:383] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.329015 22458 raft_consensus.cc:738] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.329293 22557 raft_consensus.cc:2267] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:00.333710 22759 raft_consensus.cc:491] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.334179 22759 raft_consensus.cc:513] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.336153 22456 ts_tablet_manager.cc:1428] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.061s user 0.006s sys 0.000s
I20250112 08:57:00.337034 22456 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:00.338526 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:00.339246 22213 raft_consensus.cc:2388] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:00.339039 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef66f5e785864d1c845f5c5a5f944473" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:00.340624 22759 leader_election.cc:290] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.340570 22002 leader_election.cc:304] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.344523 22706 raft_consensus.cc:3054] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.332867 22458 consensus_queue.cc:260] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.346730 22458 ts_tablet_manager.cc:1428] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250112 08:57:00.299036 22556 log.cc:1198] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:00.350751 22556 ts_tablet_manager.cc:1939] T 7a679595edde4a1f9b7f0916b2d760f2 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:00.352505 22557 ts_tablet_manager.cc:1905] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:00.344322 21704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.252:43161, user_credentials={real_user=slave}} blocked reactor thread for 257737us
I20250112 08:57:00.353442 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]) successfully deleted
W20250112 08:57:00.355989 21704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab -> Peer caa33dde8b5248e7ab4d6087d8e9d801 (127.21.42.252:43161): Couldn't send request to peer caa33dde8b5248e7ab4d6087d8e9d801. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250112 08:57:00.358764 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_fb55523682de4031861874aef9d6fa66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:00.360224 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb55523682de4031861874aef9d6fa66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:00.361965 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 89) took 1505 ms. Trace:
W20250112 08:57:00.374310 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e19979ef29944405b833bd6dad6f3664 consensus state has no leader
I20250112 08:57:00.362280 22556 rpcz_store.cc:276] 0112 08:56:58.856628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.856762 (+ 134us) service_pool.cc:225] Handling call
0112 08:56:59.825676 (+968914us) ts_tablet_manager.cc:1145] Deleting tablet 7a679595edde4a1f9b7f0916b2d760f2
0112 08:56:59.825780 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:56:59.966914 (+141134us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.298438 (+331524us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.361875 (+ 63437us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"tablet-delete.queue_time_us":968356}
I20250112 08:57:00.377734 22456 tablet_bootstrap.cc:654] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.379796 22572 hybrid_clock.cc:648] HybridClock initialized: now 1736672220379719 us; error 67 us; skew 500 ppm
I20250112 08:57:00.380908 22572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:00.384526 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25]) successfully deleted
I20250112 08:57:00.389554 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 88) took 1674 ms. Trace:
I20250112 08:57:00.389880 22556 rpcz_store.cc:276] 0112 08:56:58.715306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.907009 (+191703us) service_pool.cc:225] Handling call
0112 08:57:00.388774 (+1481765us) ts_tablet_manager.cc:1145] Deleting tablet dabc37f08d1948118b98ba25b689bea4
0112 08:57:00.389189 (+ 415us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:00.389495 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11008,"tablet-delete.queue_time_us":1481174}
W20250112 08:57:00.391010 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet dabc37f08d1948118b98ba25b689bea4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dabc37f08d1948118b98ba25b689bea4 already in progress: creating tablet
I20250112 08:57:00.351462 22706 raft_consensus.cc:2743] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.398461 22458 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.400007 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 94) took 1034 ms. Trace:
I20250112 08:57:00.400300 22556 rpcz_store.cc:276] 0112 08:56:59.365713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.365843 (+ 130us) service_pool.cc:225] Handling call
0112 08:57:00.399678 (+1033835us) ts_tablet_manager.cc:1145] Deleting tablet aa641fe365844c9eaa6ffcbf548d82f7
0112 08:57:00.399737 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:00.399944 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1033270}
W20250112 08:57:00.401352 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet aa641fe365844c9eaa6ffcbf548d82f7 because the tablet was not found. No further retry: Not found: Tablet not found: aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:57:00.404026 22706 raft_consensus.cc:491] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.404453 22706 raft_consensus.cc:513] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.407146 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:00.408066 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
W20250112 08:57:00.408325 22002 leader_election.cc:343] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.408802 22213 raft_consensus.cc:2463] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:00.410038 21999 leader_election.cc:304] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.410856 22706 leader_election.cc:290] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.411624 22706 raft_consensus.cc:2798] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.412032 22706 raft_consensus.cc:491] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.412392 22706 raft_consensus.cc:3054] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.416435 22461 tablet_bootstrap.cc:492] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:00.417044 22461 ts_tablet_manager.cc:1397] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.302s user 0.021s sys 0.000s
I20250112 08:57:00.417232 22556 tablet_replica.cc:331] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:00.422485 22572 webserver.cc:458] Webserver started at http://127.21.42.252:36567/ using document root <none> and password file <none>
I20250112 08:57:00.423281 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6" } modify_external_catalogs: true
I20250112 08:57:00.423823 22572 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:00.423854 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6" } modify_external_catalogs: true
I20250112 08:57:00.424131 22572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:00.432376 22556 raft_consensus.cc:2238] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:00.432842 22556 raft_consensus.cc:2267] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:00.436368 22556 ts_tablet_manager.cc:1905] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:00.441450 22710 raft_consensus.cc:491] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.441900 22710 raft_consensus.cc:513] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.445185 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:00.445676 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:00.445804 22068 raft_consensus.cc:2463] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:00.446278 22213 raft_consensus.cc:2463] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:00.443728 22710 leader_election.cc:290] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.446869 22290 leader_election.cc:304] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:00.447558 22710 raft_consensus.cc:2798] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.447863 22710 raft_consensus.cc:491] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.448177 22710 raft_consensus.cc:3054] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.448501 22461 raft_consensus.cc:357] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.450551 22458 tablet_bootstrap.cc:654] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.454337 22710 raft_consensus.cc:513] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.457038 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:00.457056 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1494ad860b76492cbcbb83c5341729dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:00.457604 22068 raft_consensus.cc:3054] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.458098 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003f1f1414584353a86214d27401291e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
W20250112 08:57:00.458508 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1494ad860b76492cbcbb83c5341729dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:00.458635 22213 raft_consensus.cc:3054] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.459139 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 2e62435dce1441a3b2fee50518662d48 (DEFAULT_TABLE table=default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6 [id=ce7c89cc8d89473cada34608b03819a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.460219 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e62435dce1441a3b2fee50518662d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.465030 22213 raft_consensus.cc:2463] T 003f1f1414584353a86214d27401291e P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:00.466050 22290 leader_election.cc:304] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:00.466373 21711 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:00.466797 22716 raft_consensus.cc:2798] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:00.467278 22068 raft_consensus.cc:2463] T 003f1f1414584353a86214d27401291e P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:00.467738 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 2e62435dce1441a3b2fee50518662d48 (DEFAULT_TABLE table=default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6 [id=ce7c89cc8d89473cada34608b03819a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.467938 22716 raft_consensus.cc:695] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:00.468868 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e62435dce1441a3b2fee50518662d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.468691 22716 consensus_queue.cc:237] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.473845 22710 leader_election.cc:290] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.474725 22759 raft_consensus.cc:491] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.475121 22759 raft_consensus.cc:513] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.478585 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.479840 22002 leader_election.cc:343] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.482353 22759 leader_election.cc:290] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.485057 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:00.485745 22213 raft_consensus.cc:2463] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:00.486707 21999 leader_election.cc:304] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.490054 22042 tablet_service.cc:1467] Processing CreateTablet for tablet 80c6ea87589249739461e629eee5f40b (DEFAULT_TABLE table=default.Table_bda308e111924297bc1d064e44647a8c [id=78bc9e684bde43269ebdb804dd657e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.491083 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c6ea87589249739461e629eee5f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.496229 22736 raft_consensus.cc:491] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.496707 22736 raft_consensus.cc:513] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.498616 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 2e62435dce1441a3b2fee50518662d48
I20250112 08:57:00.498608 22736 leader_election.cc:290] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:00.501726 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:00.502408 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e62435dce1441a3b2fee50518662d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:00 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:00.502904 21716 catalog_manager.cc:6596] ReplaceTablet: tablet 06717cb567a54ee6a268c3be5d8786ec deleted and replaced by tablet db58dd0f600f403a91b54707703abd4b
I20250112 08:57:00.503635 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.505357 22149 leader_election.cc:343] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.507510 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_be70d2c098e64ac4ac3f2fef255d0a3d" } new_table_name: "default.Table_76c1e51c857948f0b9bf826ca8c4dcd5" modify_external_catalogs: true
I20250112 08:57:00.508154 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:00.511741 22557 ts_tablet_manager.cc:1918] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:00.512116 22557 log.cc:1198] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/6ffbab945abd4fa58259a65e4fadd622
I20250112 08:57:00.512763 22557 ts_tablet_manager.cc:1939] T 6ffbab945abd4fa58259a65e4fadd622 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:00.512738 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e62435dce1441a3b2fee50518662d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:00 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:00.514169 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:00.514533 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 95) took 2019 ms. Trace:
I20250112 08:57:00.514803 22557 rpcz_store.cc:276] 0112 08:56:58.495250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.496521 (+ 1271us) service_pool.cc:225] Handling call
0112 08:57:00.327626 (+1831105us) ts_tablet_manager.cc:1145] Deleting tablet 6ffbab945abd4fa58259a65e4fadd622
0112 08:57:00.327668 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:00.414396 (+ 86728us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.511539 (+ 97143us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.514479 (+ 2940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830339}
I20250112 08:57:00.516062 22736 raft_consensus.cc:491] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.516474 22736 raft_consensus.cc:513] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.518646 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 6ffbab945abd4fa58259a65e4fadd622 (table default.Table_abb85608f59e40c1bcb4004eb5783cb6 [id=8bb37e8f837d4c5299debd19da8416b7]) successfully deleted
W20250112 08:57:00.519292 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 2e62435dce1441a3b2fee50518662d48 because the tablet was not found. No further retry: Not found: Tablet not found: 2e62435dce1441a3b2fee50518662d48
I20250112 08:57:00.522894 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:00.523972 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ce6f55c444fe0852ad72cc7383cc1" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:00.525578 22146 leader_election.cc:343] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.525967 22146 leader_election.cc:304] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.526638 22695 raft_consensus.cc:2743] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:00.526995 22146 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.531881 22557 tablet_replica.cc:331] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:00.532625 22557 raft_consensus.cc:2238] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:00.533057 22557 raft_consensus.cc:2267] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:00.534591 22557 ts_tablet_manager.cc:1905] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:00.537212 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:00.538537 21715 catalog_manager.cc:6596] ReplaceTablet: tablet 010ce6f55c444fe0852ad72cc7383cc1 deleted and replaced by tablet 72e8f51e50994929b40a5e0679b07d8c
I20250112 08:57:00.540199 22760 raft_consensus.cc:491] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.540688 22760 raft_consensus.cc:513] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.542652 22760 leader_election.cc:290] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.544030 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c7056aaf8844da915acec5f9f92eeb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.544570 21999 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb
W20250112 08:57:00.545742 22002 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.546146 22461 raft_consensus.cc:383] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.546072 22002 leader_election.cc:304] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.546422 22461 raft_consensus.cc:738] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.547209 22666 raft_consensus.cc:2743] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.547088 22461 consensus_queue.cc:260] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.547670 22666 raft_consensus.cc:491] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.548058 22666 raft_consensus.cc:513] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.549700 22666 leader_election.cc:290] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.551206 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:00.551549 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:00.552179 22212 raft_consensus.cc:2463] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
W20250112 08:57:00.552654 22002 leader_election.cc:343] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.542413 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 69651ab160db41b3a187542ac1cd2839 (table default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]). Expected version 1 got 0
I20250112 08:57:00.553539 21999 leader_election.cc:304] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.554608 22760 raft_consensus.cc:2798] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.554924 22760 raft_consensus.cc:491] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.555198 22760 raft_consensus.cc:3054] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.560494 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:00.560652 22556 ts_tablet_manager.cc:1918] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:00.561005 22556 log.cc:1198] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:57:00.561601 22556 ts_tablet_manager.cc:1939] T 69a9c2b5a5d044c789a3eda07e8a312d P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:00.561950 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:00.566404 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:00.577454 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ce6f55c444fe0852ad72cc7383cc1" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.578877 22149 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.579288 22149 leader_election.cc:304] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.579965 22695 raft_consensus.cc:2743] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.580085 22736 leader_election.cc:290] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:00.581383 22716 consensus_queue.cc:1035] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:00.587244 22710 consensus_queue.cc:1035] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:00.589251 22461 ts_tablet_manager.cc:1428] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.172s user 0.004s sys 0.002s
I20250112 08:57:00.487716 22759 raft_consensus.cc:2798] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.589804 22759 raft_consensus.cc:491] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.595835 22456 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:00.596244 22456 ts_tablet_manager.cc:1397] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.259s user 0.009s sys 0.001s
I20250112 08:57:00.598505 22456 raft_consensus.cc:357] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.599187 22456 raft_consensus.cc:383] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.599427 22456 raft_consensus.cc:738] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.600618 22456 consensus_queue.cc:260] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.602681 22456 ts_tablet_manager.cc:1428] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:00.603523 22456 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:00.606650 22759 raft_consensus.cc:3054] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.611487 22461 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:00.612160 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 95) took 1245 ms. Trace:
I20250112 08:57:00.612473 22556 rpcz_store.cc:276] 0112 08:56:59.366923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.367024 (+ 101us) service_pool.cc:225] Handling call
0112 08:57:00.417005 (+1049981us) ts_tablet_manager.cc:1145] Deleting tablet 69a9c2b5a5d044c789a3eda07e8a312d
0112 08:57:00.417084 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:00.504760 (+ 87676us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.560163 (+ 55403us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.612104 (+ 51941us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13151,"tablet-delete.queue_time_us":1049435}
I20250112 08:57:00.614413 22458 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.614826 22458 ts_tablet_manager.cc:1397] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.000s
I20250112 08:57:00.616421 22329 tablet_service.cc:1467] Processing CreateTablet for tablet 2e62435dce1441a3b2fee50518662d48 (DEFAULT_TABLE table=default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6 [id=ce7c89cc8d89473cada34608b03819a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.617822 22329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e62435dce1441a3b2fee50518662d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.624095 22458 raft_consensus.cc:357] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.624769 22458 raft_consensus.cc:383] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.624998 22458 raft_consensus.cc:738] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.625427 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e62435dce1441a3b2fee50518662d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:00 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:57:00.565470 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 06717cb567a54ee6a268c3be5d8786ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06717cb567a54ee6a268c3be5d8786ec already in progress: creating tablet
W20250112 08:57:00.626644 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 61206.4us
I20250112 08:57:00.629592 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:00.631013 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:00.632962 22458 consensus_queue.cc:260] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.636773 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 3 got 0
I20250112 08:57:00.637764 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 69a9c2b5a5d044c789a3eda07e8a312d (table default.Table_fad3e6f7b8f84d748b8429927a55070a [id=cb3251ba8e1944d2b91609025311b09d]) successfully deleted
W20250112 08:57:00.638451 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 35f60d1f4bed4721b73deb9ba787417c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f60d1f4bed4721b73deb9ba787417c already in progress: creating tablet
W20250112 08:57:00.639036 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 2e62435dce1441a3b2fee50518662d48 because the tablet was not found. No further retry: Not found: Tablet not found: 2e62435dce1441a3b2fee50518662d48
I20250112 08:57:00.644923 22736 raft_consensus.cc:491] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.644774 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8e00eee0aedf49bcbf1528e033fa3620" } new_table_name: "default.Table_66ec2be61ba0470195e2c535112eb097" modify_external_catalogs: true
I20250112 08:57:00.645417 22736 raft_consensus.cc:513] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.647001 22458 ts_tablet_manager.cc:1428] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250112 08:57:00.650646 22695 raft_consensus.cc:491] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.651136 22695 raft_consensus.cc:513] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.653326 22695 leader_election.cc:290] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:00.644800 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:00.654320 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06717cb567a54ee6a268c3be5d8786ec" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:00.655460 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 422e1484cd84479abba824cc8a6fae1f (table default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]). Expected version 1 got 0
I20250112 08:57:00.656059 22458 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:57:00.656432 22146 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.656904 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e19979ef29944405b833bd6dad6f3664 (table default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf]). Expected version 2 got 0
I20250112 08:57:00.658653 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06717cb567a54ee6a268c3be5d8786ec" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.659823 22149 leader_election.cc:343] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: 7a679595edde4a1f9b7f0916b2d760f2
W20250112 08:57:00.660665 22149 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.660995 22149 leader_election.cc:304] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.661331 22710 raft_consensus.cc:491] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:00.661895 22695 raft_consensus.cc:2743] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.661770 22710 raft_consensus.cc:513] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:00.637777 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 010ce6f55c444fe0852ad72cc7383cc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010ce6f55c444fe0852ad72cc7383cc1 already in progress: creating tablet
I20250112 08:57:00.663579 22710 leader_election.cc:290] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.664577 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 80c6ea87589249739461e629eee5f40b (DEFAULT_TABLE table=default.Table_bda308e111924297bc1d064e44647a8c [id=78bc9e684bde43269ebdb804dd657e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.665052 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090d258647b4dd7bd91cf64011735dd" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:00.665517 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c6ea87589249739461e629eee5f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.665624 22068 raft_consensus.cc:2463] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:00.666919 22666 raft_consensus.cc:491] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:57:00.668002 22290 leader_election.cc:343] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a090d258647b4dd7bd91cf64011735dd
I20250112 08:57:00.667305 22666 raft_consensus.cc:513] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.668565 22290 leader_election.cc:304] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:00.669237 22710 raft_consensus.cc:2798] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:00.669509 22710 raft_consensus.cc:491] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:00.669631 22666 leader_election.cc:290] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.669808 22710 raft_consensus.cc:3054] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.670818 22461 tablet_bootstrap.cc:654] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.671069 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_dba2962989f24aec9af818bd34ea9122"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:00.672855 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dba2962989f24aec9af818bd34ea9122 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:00.675632 22736 leader_election.cc:290] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:57:00.676611 22146 leader_election.cc:343] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:00.677134 22146 leader_election.cc:304] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.678141 22736 raft_consensus.cc:2743] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.678189 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:00.679625 22002 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.680303 22458 tablet_bootstrap.cc:654] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.680357 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
W20250112 08:57:00.681556 21999 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Illegal state: Tablet not RUNNING: STOPPED
I20250112 08:57:00.681974 21999 leader_election.cc:304] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:00.684023 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 815d246b7d414aadbb69ac692a5d40c7 consensus state has no leader
I20250112 08:57:00.685009 22666 raft_consensus.cc:2743] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:00.685197 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_ec36ed57a80f4350bda6894141741ba3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:00.686846 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec36ed57a80f4350bda6894141741ba3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:00.687441 22461 tablet_bootstrap.cc:492] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:00.687857 22461 ts_tablet_manager.cc:1397] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.003s
I20250112 08:57:00.689843 22760 raft_consensus.cc:513] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.690886 22461 raft_consensus.cc:357] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.691576 22461 raft_consensus.cc:383] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.687863 22710 raft_consensus.cc:513] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.694599 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a090d258647b4dd7bd91cf64011735dd" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:00.694713 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:00.695186 22068 raft_consensus.cc:3054] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:00.696635 22002 leader_election.cc:343] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.696954 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd402674080464cb3bde3697ac23180 (DEFAULT_TABLE table=default.Table_f43c29fa6315404ca33380d0034bf1b1 [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.698554 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cd402674080464cb3bde3697ac23180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.701455 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd402674080464cb3bde3697ac23180 (DEFAULT_TABLE table=default.Table_f43c29fa6315404ca33380d0034bf1b1 [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.702533 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cd402674080464cb3bde3697ac23180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.698557 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:00.703873 22213 raft_consensus.cc:3054] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:00.704627 22068 raft_consensus.cc:2463] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:00.705057 22047 tablet_service.cc:1467] Processing CreateTablet for tablet ea15d9405b9944db8e00121965b9ffde (DEFAULT_TABLE table=default.Table_250ba1babd9c419e8933849397cdff93 [id=a42ff5d5b1f949ef97f131233f826798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.705950 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea15d9405b9944db8e00121965b9ffde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.705727 22290 leader_election.cc:304] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:00.706507 22716 raft_consensus.cc:2798] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:00.706979 22716 raft_consensus.cc:695] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:00.707682 22716 consensus_queue.cc:237] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.691893 22461 raft_consensus.cc:738] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.716990 22760 leader_election.cc:290] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.718048 22190 tablet_service.cc:1467] Processing CreateTablet for tablet ea15d9405b9944db8e00121965b9ffde (DEFAULT_TABLE table=default.Table_250ba1babd9c419e8933849397cdff93 [id=a42ff5d5b1f949ef97f131233f826798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.718161 22046 tablet_service.cc:1467] Processing CreateTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (DEFAULT_TABLE table=default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.719014 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea15d9405b9944db8e00121965b9ffde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.719123 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:00.720185 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53167910ab0a4accb1d1615d537ba7ae consensus state has no leader
I20250112 08:57:00.720659 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 98ddac23210248e3be1aba6591641b2d (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.720630 22187 tablet_service.cc:1467] Processing CreateTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (DEFAULT_TABLE table=default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.721644 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ddac23210248e3be1aba6591641b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.722049 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 98ddac23210248e3be1aba6591641b2d (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.721943 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_372c2c8732cb429f9a5f0510ea2caca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:00.723573 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_372c2c8732cb429f9a5f0510ea2caca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:00.724479 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93c5e0d2a14c16a8080b21da64eb85 (DEFAULT_TABLE table=default.Table_e85c484e045548ada7a5fbc4545abbd0 [id=38a60a3b2406402bb8eeae121aa205c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.725313 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93c5e0d2a14c16a8080b21da64eb85. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:00.726763 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 815d246b7d414aadbb69ac692a5d40c7 consensus state has no leader
I20250112 08:57:00.728307 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ddac23210248e3be1aba6591641b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.729583 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.744774 22572 fs_manager.cc:714] Time spent opening directory manager: real 0.258s user 0.008s sys 0.027s
I20250112 08:57:00.749557 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_c18e0bb841d942978709c4d445929483" } modify_external_catalogs: true
W20250112 08:57:00.750931 22290 leader_election.cc:343] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a090d258647b4dd7bd91cf64011735dd
I20250112 08:57:00.751924 22461 consensus_queue.cc:260] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.753089 22710 leader_election.cc:290] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.768649 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_c18e0bb841d942978709c4d445929483" } modify_external_catalogs: true
I20250112 08:57:00.761792 22461 ts_tablet_manager.cc:1428] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.074s user 0.004s sys 0.001s
I20250112 08:57:00.772850 22461 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:00.776181 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 06717cb567a54ee6a268c3be5d8786ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06717cb567a54ee6a268c3be5d8786ec already in progress: creating tablet
W20250112 08:57:00.776863 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 010ce6f55c444fe0852ad72cc7383cc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010ce6f55c444fe0852ad72cc7383cc1 already in progress: creating tablet
I20250112 08:57:00.778517 22213 raft_consensus.cc:2463] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:00.778774 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 80c6ea87589249739461e629eee5f40b (DEFAULT_TABLE table=default.Table_bda308e111924297bc1d064e44647a8c [id=78bc9e684bde43269ebdb804dd657e65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.779831 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c6ea87589249739461e629eee5f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.779636 21999 leader_election.cc:304] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:00.779637 22338 tablet_service.cc:1467] Processing CreateTablet for tablet ea15d9405b9944db8e00121965b9ffde (DEFAULT_TABLE table=default.Table_250ba1babd9c419e8933849397cdff93 [id=a42ff5d5b1f949ef97f131233f826798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.780647 22760 raft_consensus.cc:2798] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:00.780800 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea15d9405b9944db8e00121965b9ffde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.781122 22557 ts_tablet_manager.cc:1918] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:00.781431 22557 log.cc:1198] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:00.781405 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd402674080464cb3bde3697ac23180 (DEFAULT_TABLE table=default.Table_f43c29fa6315404ca33380d0034bf1b1 [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.781992 22557 ts_tablet_manager.cc:1939] T dabc37f08d1948118b98ba25b689bea4 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:00.782723 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cd402674080464cb3bde3697ac23180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.783691 22760 raft_consensus.cc:695] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:00.784456 22760 consensus_queue.cc:237] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.805491 22458 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.806178 22458 ts_tablet_manager.cc:1397] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.159s user 0.010s sys 0.001s
I20250112 08:57:00.810417 22784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:00.811398 22759 raft_consensus.cc:513] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.811887 22458 raft_consensus.cc:357] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.812538 22458 raft_consensus.cc:383] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.812794 22458 raft_consensus.cc:738] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.815035 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:00.817576 22461 tablet_bootstrap.cc:654] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.820003 22572 fs_manager.cc:730] Time spent opening block manager: real 0.072s user 0.009s sys 0.013s
W20250112 08:57:00.816593 22002 leader_election.cc:343] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:00.817107 21999 consensus_peers.cc:487] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c683c8c64a441e8b5be52b077fdcb08. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:00.822337 22760 consensus_queue.cc:1035] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:00.823168 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 96) took 1895 ms. Trace:
I20250112 08:57:00.820451 22572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
uuid: "caa33dde8b5248e7ab4d6087d8e9d801"
format_stamp: "Formatted at 2025-01-12 08:56:42 on dist-test-slave-ghv6"
I20250112 08:57:00.823515 22557 rpcz_store.cc:276] 0112 08:56:58.927977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.928139 (+ 162us) service_pool.cc:225] Handling call
0112 08:57:00.531728 (+1603589us) ts_tablet_manager.cc:1145] Deleting tablet dabc37f08d1948118b98ba25b689bea4
0112 08:57:00.531775 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:00.629215 (+ 97440us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.780915 (+151700us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:00.823108 (+ 42193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602915}
I20250112 08:57:00.825721 22572 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:00.826273 22557 tablet_replica.cc:331] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:00.826961 22557 raft_consensus.cc:2238] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:00.827411 22557 raft_consensus.cc:2267] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:00.834875 22456 tablet_bootstrap.cc:654] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.838083 22458 consensus_queue.cc:260] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.844214 22458 ts_tablet_manager.cc:1428] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.038s user 0.001s sys 0.004s
I20250112 08:57:00.845299 22759 leader_election.cc:290] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:00.851166 22458 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.875119 22458 tablet_bootstrap.cc:654] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.882277 22461 tablet_bootstrap.cc:492] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:00.882676 22461 ts_tablet_manager.cc:1397] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.007s
I20250112 08:57:00.887794 21715 catalog_manager.cc:5526] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:00.889380 22461 raft_consensus.cc:357] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.890030 22461 raft_consensus.cc:383] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.890278 22461 raft_consensus.cc:738] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.890044 21715 catalog_manager.cc:5526] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:00.891525 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 69651ab160db41b3a187542ac1cd2839 (table default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]). Expected version 1 got 0
I20250112 08:57:00.892771 21715 catalog_manager.cc:5526] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:00.894356 22458 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:00.894757 22458 ts_tablet_manager.cc:1397] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20250112 08:57:00.896083 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 10bf912f189a42c19deb744c1c9f4daf
I20250112 08:57:00.897490 21712 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56650 (request call id 143) took 2152 ms. Trace:
I20250112 08:57:00.899626 22461 consensus_queue.cc:260] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.901495 22458 raft_consensus.cc:357] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.902130 22458 raft_consensus.cc:383] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:00.902366 22458 raft_consensus.cc:738] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:00.908653 22461 ts_tablet_manager.cc:1428] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
W20250112 08:57:00.907569 21709 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 32) took 1709 ms (client timeout 1000 ms). Trace:
I20250112 08:57:00.909416 22458 consensus_queue.cc:260] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.913506 22461 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:00.919066 21711 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 32) took 1536 ms (client timeout 1000 ms). Trace:
W20250112 08:57:00.919442 21711 rpcz_store.cc:269] 0112 08:56:59.382876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.382993 (+ 117us) service_pool.cc:225] Handling call
0112 08:57:00.918990 (+1535997us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":12}
I20250112 08:57:00.919788 22458 ts_tablet_manager.cc:1428] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250112 08:57:00.897836 21712 rpcz_store.cc:276] 0112 08:56:58.745136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.752567 (+ 7431us) service_pool.cc:225] Handling call
0112 08:56:58.920680 (+168113us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26574616 00007fcf242dcb44 00007fcf242dcad2 00007fcf242dca01 00007fcf242dc97c 00007fcf242dbc32 00007fcf2259adfc 00007fcf2259a236
0112 08:56:58.933974 (+ 13294us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:56:59.339768 (+405794us) catalog_manager.cc:2753] Modifying in-memory table state
0112 08:56:59.339935 (+ 167us) catalog_manager.cc:2762] Locking tablets
0112 08:56:59.816092 (+476157us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 08:57:00.365994 (+549902us) catalog_manager.cc:2794] Removing table from by-name map
0112 08:57:00.366296 (+ 302us) catalog_manager.cc:2810] Committing in-memory state
0112 08:57:00.895992 (+529696us) catalog_manager.cc:2814] Aborting table tasks
0112 08:57:00.897419 (+ 1427us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:00.064447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:00.088607 (+ 24160us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:00.088632 (+ 25us) write_op.cc:435] Acquired schema lock
0112 08:57:00.088645 (+ 13us) tablet.cc:659] Decoding operations
0112 08:57:00.088956 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:00.088988 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:00.089009 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:57:00.089237 (+ 228us) tablet.cc:698] Row locks acquired
0112 08:57:00.089262 (+ 25us) write_op.cc:260] PREPARE: finished
0112 08:57:00.089364 (+ 102us) write_op.cc:270] Start()
0112 08:57:00.089438 (+ 74us) write_op.cc:276] Timestamp: P: 1736672220089314 usec, L: 0
0112 08:57:00.089452 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:57:00.089896 (+ 444us) log.cc:844] Serialized 771 byte log entry
0112 08:57:00.354478 (+264582us) op_driver.cc:464] REPLICATION: finished
0112 08:57:00.357017 (+ 2539us) write_op.cc:301] APPLY: starting
0112 08:57:00.357068 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:00.357158 (+ 90us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:00.357253 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:00.357502 (+ 249us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:00.357629 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 08:57:00.357671 (+ 42us) write_op.cc:312] APPLY: finished
0112 08:57:00.357854 (+ 183us) log.cc:844] Serialized 29 byte log entry
0112 08:57:00.357997 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:00.358064 (+ 67us) write_op.cc:454] Released schema lock
0112 08:57:00.358096 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:00.367026 (+ 8930us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:00.367058 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":383025024,"child_traces":[["op",{"apply.queue_time_us":2411,"apply.run_cpu_time_us":1293,"apply.run_wall_time_us":9959,"num_ops":2,"prepare.queue_time_us":246451,"prepare.run_cpu_time_us":2013,"prepare.run_wall_time_us":116169,"raft.queue_time_us":104,"raft.run_cpu_time_us":955,"raft.run_wall_time_us":954,"replication_time_us":264924,"spinlock_wait_cycles":69888}]]}
W20250112 08:57:00.909365 21709 rpcz_store.cc:269] 0112 08:56:59.198253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.198356 (+ 103us) service_pool.cc:225] Handling call
0112 08:57:00.907501 (+1709145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:56:59.774759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:56:59.775076 (+ 317us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:56:59.775098 (+ 22us) write_op.cc:435] Acquired schema lock
0112 08:56:59.775110 (+ 12us) tablet.cc:659] Decoding operations
0112 08:56:59.775338 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:56:59.775373 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0112 08:56:59.775393 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:56:59.775527 (+ 134us) tablet.cc:698] Row locks acquired
0112 08:56:59.775540 (+ 13us) write_op.cc:260] PREPARE: finished
0112 08:56:59.775632 (+ 92us) write_op.cc:270] Start()
0112 08:56:59.775708 (+ 76us) write_op.cc:276] Timestamp: P: 1736672219775573 usec, L: 0
0112 08:56:59.775721 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 08:56:59.776175 (+ 454us) log.cc:844] Serialized 533 byte log entry
0112 08:56:59.812550 (+ 36375us) op_driver.cc:464] REPLICATION: finished
0112 08:56:59.812860 (+ 310us) write_op.cc:301] APPLY: starting
0112 08:56:59.812916 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0112 08:56:59.813007 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0112 08:56:59.813102 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:56:59.813269 (+ 167us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:56:59.813387 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:56:59.813437 (+ 50us) write_op.cc:312] APPLY: finished
0112 08:56:59.813621 (+ 184us) log.cc:844] Serialized 23 byte log entry
0112 08:56:59.813747 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:59.813807 (+ 60us) write_op.cc:454] Released schema lock
0112 08:56:59.813855 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0112 08:56:59.873194 (+ 59339us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:56:59.873219 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1162,"apply.run_wall_time_us":1162,"num_ops":1,"prepare.queue_time_us":441506,"prepare.run_cpu_time_us":1779,"prepare.run_wall_time_us":98397,"raft.queue_time_us":10606,"raft.run_cpu_time_us":687,"raft.run_wall_time_us":1546,"replication_time_us":36734}]]}
I20250112 08:57:00.925688 22458 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:00.934808 22461 tablet_bootstrap.cc:654] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:00.936060 21717 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 33) took 1381 ms (client timeout 1000 ms). Trace:
W20250112 08:57:00.936527 21717 rpcz_store.cc:269] 0112 08:56:59.554327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.672982 (+118655us) service_pool.cc:225] Handling call
0112 08:57:00.935971 (+1262989us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":10}
I20250112 08:57:00.939818 22458 tablet_bootstrap.cc:654] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:00.951987 22335 tablet_service.cc:1467] Processing CreateTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (DEFAULT_TABLE table=default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.952507 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 98ddac23210248e3be1aba6591641b2d (DEFAULT_TABLE table=default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.952940 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.953408 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ddac23210248e3be1aba6591641b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.957003 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93c5e0d2a14c16a8080b21da64eb85 (DEFAULT_TABLE table=default.Table_e85c484e045548ada7a5fbc4545abbd0 [id=38a60a3b2406402bb8eeae121aa205c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:00.957976 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93c5e0d2a14c16a8080b21da64eb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:00.592124 22706 raft_consensus.cc:513] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:00.967577 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:00.967854 22706 leader_election.cc:290] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:00.968973 22002 leader_election.cc:343] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:00.976570 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf912f189a42c19deb744c1c9f4daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:00.977222 22565 tablet_replica.cc:331] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:00.977885 22565 raft_consensus.cc:2238] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:00.978339 22565 raft_consensus.cc:2267] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:00.980108 22565 ts_tablet_manager.cc:1905] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:00.980662 22461 tablet_bootstrap.cc:492] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:00.981042 22461 ts_tablet_manager.cc:1397] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.000s
I20250112 08:57:01.000162 21710 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet db58dd0f600f403a91b54707703abd4b
I20250112 08:57:00.837320 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:00.855300 22557 ts_tablet_manager.cc:1905] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:00.866708 22736 raft_consensus.cc:491] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.003657 22213 raft_consensus.cc:3054] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.003762 22736 raft_consensus.cc:513] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.005671 22736 leader_election.cc:290] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:00.973412 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:01.006197 22212 raft_consensus.cc:3054] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.009995 22213 raft_consensus.cc:2463] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:01.011416 22706 raft_consensus.cc:491] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.011708 21999 leader_election.cc:304] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:01.011803 22706 raft_consensus.cc:513] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.012739 22759 raft_consensus.cc:2798] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:01.013125 22759 raft_consensus.cc:695] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:00.974133 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93c5e0d2a14c16a8080b21da64eb85 (DEFAULT_TABLE table=default.Table_e85c484e045548ada7a5fbc4545abbd0 [id=38a60a3b2406402bb8eeae121aa205c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.013804 22759 consensus_queue.cc:237] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.014271 22706 leader_election.cc:290] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.014761 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d93c5e0d2a14c16a8080b21da64eb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.017933 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.018709 22211 raft_consensus.cc:2463] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:00.975294 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf912f189a42c19deb744c1c9f4daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.019812 21999 leader_election.cc:304] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:01.020576 22760 raft_consensus.cc:2798] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:01.021261 22760 raft_consensus.cc:491] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:01.021659 22760 raft_consensus.cc:3054] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.022819 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:01.024457 22146 leader_election.cc:343] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.029346 22760 raft_consensus.cc:513] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.030534 22557 ts_tablet_manager.cc:1918] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.030864 22557 log.cc:1198] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:01.041180 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.041508 22557 ts_tablet_manager.cc:1939] T 7a679595edde4a1f9b7f0916b2d760f2 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:01.042006 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ffbab945abd4fa58259a65e4fadd622 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:58 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.042863 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:01.043398 22211 raft_consensus.cc:3054] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.043998 22760 leader_election.cc:290] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.044740 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.044991 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 97) took 2116 ms. Trace:
I20250112 08:57:01.045258 22557 rpcz_store.cc:276] 0112 08:56:58.928312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:58.928994 (+ 682us) service_pool.cc:225] Handling call
0112 08:57:00.826130 (+1897136us) ts_tablet_manager.cc:1145] Deleting tablet 7a679595edde4a1f9b7f0916b2d760f2
0112 08:57:00.826172 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.018142 (+191970us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.030332 (+ 12190us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.044928 (+ 14596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896544}
I20250112 08:57:01.049171 22212 raft_consensus.cc:2463] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:01.050506 21999 leader_election.cc:304] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:01.052845 22557 tablet_replica.cc:331] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:01.053596 22557 raft_consensus.cc:2238] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.054041 22557 raft_consensus.cc:2267] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.059369 22760 raft_consensus.cc:2798] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:01.052181 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.060217 22760 raft_consensus.cc:695] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:01.061787 22760 consensus_queue.cc:237] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.065124 22736 raft_consensus.cc:491] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.066984 22211 raft_consensus.cc:2463] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:01.057395 22557 ts_tablet_manager.cc:1905] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.068253 21999 leader_election.cc:304] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:01.068961 22666 raft_consensus.cc:491] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.069344 22666 raft_consensus.cc:513] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.071621 22666 leader_election.cc:290] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.073603 22666 raft_consensus.cc:2798] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:01.074051 22666 raft_consensus.cc:695] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:01.074856 22666 consensus_queue.cc:237] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.075750 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6632e6bd7efc4fecb864bea93b604e87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:01.065577 22736 raft_consensus.cc:513] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.080080 22736 leader_election.cc:290] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
W20250112 08:57:01.080132 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6632e6bd7efc4fecb864bea93b604e87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.080514 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f60d1f4bed4721b73deb9ba787417c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:01.080834 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.081427 22212 raft_consensus.cc:2463] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
W20250112 08:57:01.081920 22146 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.082504 21999 leader_election.cc:304] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:01.083310 22706 raft_consensus.cc:2798] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:01.083707 22706 raft_consensus.cc:491] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:01.083827 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_fb55523682de4031861874aef9d6fa66" } new_table_name: "default.Table_15a31e5fe0e74322bcbefd51437a08b7" modify_external_catalogs: true
I20250112 08:57:01.084089 22706 raft_consensus.cc:3054] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.085956 22335 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf912f189a42c19deb744c1c9f4daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.087386 21710 master_service.cc:946] ReplaceTablet: received request to replace tablet a44cad3bd5a84e75a41733d8d2f345a8 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:01.090040 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet dabc37f08d1948118b98ba25b689bea4 (table default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6 [id=dc7a2e8940134ac184a6434f029cd95d]) successfully deleted
I20250112 08:57:01.096014 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet aa641fe365844c9eaa6ffcbf548d82f7 with delete_type TABLET_DATA_DELETED (replaced by tablet a44cad3bd5a84e75a41733d8d2f345a8) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.090960 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 7a679595edde4a1f9b7f0916b2d760f2 (table default.Table_3f79f09bbc3949a68c55ec5dab776291 [id=adee3342aa1141818220e48211f1bf25]) successfully deleted
I20250112 08:57:01.101400 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:01.102005 22357 raft_consensus.cc:3054] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.104394 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.105326 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:01.091009 22706 raft_consensus.cc:513] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.110885 22458 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
W20250112 08:57:01.110977 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 69a9c2b5a5d044c789a3eda07e8a312d because the tablet was not found. No further retry: Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:57:01.111111 22331 tablet_service.cc:1514] Processing DeleteTablet for tablet f252f0ec641d4bf5b4fe1399a8ced51e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.111364 22458 ts_tablet_manager.cc:1397] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.001s
I20250112 08:57:01.111855 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c683c8c64a441e8b5be52b077fdcb08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:54 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.112630 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:57 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.112828 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:01.113407 22211 raft_consensus.cc:3054] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.113843 22458 raft_consensus.cc:357] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.114475 22458 raft_consensus.cc:383] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.116626 22565 ts_tablet_manager.cc:1918] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:57:01.117031 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.119565 22211 raft_consensus.cc:2463] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
W20250112 08:57:01.121126 21708 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 33) took 811 ms (client timeout 1000 ms). Trace:
I20250112 08:57:01.123761 22354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:01.129045 21708 rpcz_store.cc:269] 0112 08:57:00.310065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:00.359010 (+ 48945us) service_pool.cc:225] Handling call
0112 08:57:01.121070 (+762060us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15}
I20250112 08:57:01.131836 22456 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:01.132261 22456 ts_tablet_manager.cc:1397] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.529s user 0.012s sys 0.001s
I20250112 08:57:01.134416 22456 raft_consensus.cc:357] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.135028 22456 raft_consensus.cc:383] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.135259 22456 raft_consensus.cc:738] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.136476 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet b0192d66e64d4d39b4bcfd6565dff723 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.135812 22456 consensus_queue.cc:260] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.129673 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:01.140944 22456 ts_tablet_manager.cc:1428] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250112 08:57:01.141732 22456 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:01.144315 22556 tablet_replica.cc:331] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:01.116464 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253b5b6ea6f044fc8dcaafdc2e587f84" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:01.146072 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f60d1f4bed4721b73deb9ba787417c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:01.147444 22456 tablet_bootstrap.cc:654] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.152722 22682 consensus_queue.cc:1035] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:01.154639 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:01.156255 21710 catalog_manager.cc:6596] ReplaceTablet: tablet a44cad3bd5a84e75a41733d8d2f345a8 deleted and replaced by tablet e61e80e259db4645942141754c8491a3
I20250112 08:57:01.155938 22557 ts_tablet_manager.cc:1918] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.156662 22557 log.cc:1198] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/93be4e0fc31c429f860c1a0cec64849b
I20250112 08:57:01.157296 22557 ts_tablet_manager.cc:1939] T 93be4e0fc31c429f860c1a0cec64849b P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:01.159268 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 101) took 1776 ms. Trace:
I20250112 08:57:01.159623 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:44720
W20250112 08:57:01.162647 22290 consensus_peers.cc:487] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:01.165354 22682 consensus_queue.cc:1035] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:01.191812 22290 consensus_peers.cc:487] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7451e83f3b554936a79c6724ebe76cbb. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:01.159546 22557 rpcz_store.cc:276] 0112 08:56:59.382873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.383329 (+ 456us) service_pool.cc:225] Handling call
0112 08:57:01.052700 (+1669371us) ts_tablet_manager.cc:1145] Deleting tablet 93be4e0fc31c429f860c1a0cec64849b
0112 08:57:01.052738 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.088232 (+ 35494us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.138379 (+ 50147us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.159219 (+ 20840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668801}
I20250112 08:57:01.192943 22557 tablet_replica.cc:331] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:01.193706 22557 raft_consensus.cc:2238] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.194123 22557 raft_consensus.cc:2267] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.199529 22556 raft_consensus.cc:2238] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.200028 22556 raft_consensus.cc:2267] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.201686 22556 ts_tablet_manager.cc:1905] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.212486 22557 ts_tablet_manager.cc:1905] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.218171 22357 raft_consensus.cc:2463] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
W20250112 08:57:01.223299 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d686b08555414954aba52ea077889660 has no consensus state
W20250112 08:57:01.223531 22329 consensus_meta.cc:287] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: Time spent flushing consensus metadata: real 0.587s user 0.001s sys 0.005s
I20250112 08:57:01.226377 22461 raft_consensus.cc:357] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.227133 22461 raft_consensus.cc:383] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.120488 22706 leader_election.cc:290] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.227519 22461 raft_consensus.cc:738] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.124195 21999 leader_election.cc:304] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:01.228099 22565 log.cc:1198] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/10bf912f189a42c19deb744c1c9f4daf
W20250112 08:57:01.228463 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 104361us
I20250112 08:57:01.228844 22565 ts_tablet_manager.cc:1939] T 10bf912f189a42c19deb744c1c9f4daf P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:01.228467 22461 consensus_queue.cc:260] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.229447 22706 raft_consensus.cc:2798] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:01.229861 22706 raft_consensus.cc:695] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:01.230566 22706 consensus_queue.cc:237] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:01.126243 22002 leader_election.cc:343] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:01.233229 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 107016us
I20250112 08:57:01.233817 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 69a9c2b5a5d044c789a3eda07e8a312d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:53 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.234745 22461 ts_tablet_manager.cc:1428] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.253s user 0.005s sys 0.001s
W20250112 08:57:01.234773 22002 leader_election.cc:343] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.114711 22458 raft_consensus.cc:738] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.236311 22458 consensus_queue.cc:260] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.238358 22458 ts_tablet_manager.cc:1428] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.127s user 0.006s sys 0.001s
I20250112 08:57:01.239071 22458 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:01.239437 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.244737 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.245811 22565 tablet_replica.cc:331] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:01.246794 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a679595edde4a1f9b7f0916b2d760f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:57 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.249022 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:56484
W20250112 08:57:01.253304 21999 consensus_peers.cc:487] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:01.254730 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.257931 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.260594 22458 tablet_bootstrap.cc:654] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.260866 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:01.265414 22149 leader_election.cc:343] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.265816 22149 leader_election.cc:304] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:01.269063 22759 consensus_queue.cc:1035] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:01.270493 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 7451e83f3b554936a79c6724ebe76cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.276003 22565 raft_consensus.cc:2238] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:01.276554 22461 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:01.280781 22736 raft_consensus.cc:2743] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:01.281905 22149 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.283128 21715 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c683c8c64a441e8b5be52b077fdcb08 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:01.289005 22149 leader_election.cc:304] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:01.289503 22149 raft_consensus.cc:2681] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:57:01.303766 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ec21905b8bd64621b9687023c5d3024e" } new_table_name: "default.Table_5bd9277e65f04760b8979a42ceef3074" modify_external_catalogs: true
I20250112 08:57:01.308086 22456 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:01.308480 22456 ts_tablet_manager.cc:1397] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.167s user 0.035s sys 0.052s
I20250112 08:57:01.322232 22456 raft_consensus.cc:357] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.322912 22456 raft_consensus.cc:383] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.323148 22456 raft_consensus.cc:738] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.323644 22556 ts_tablet_manager.cc:1918] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.324024 22556 log.cc:1198] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/f252f0ec641d4bf5b4fe1399a8ced51e
I20250112 08:57:01.324695 22556 ts_tablet_manager.cc:1939] T f252f0ec641d4bf5b4fe1399a8ced51e P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
W20250112 08:57:01.329926 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 consensus state has no leader
I20250112 08:57:01.332903 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_2c562866672a4bbfa557feddf8f6464e" } modify_external_catalogs: true
I20250112 08:57:01.333447 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_2c562866672a4bbfa557feddf8f6464e" } modify_external_catalogs: true
I20250112 08:57:01.334933 22456 consensus_queue.cc:260] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.336925 22456 ts_tablet_manager.cc:1428] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250112 08:57:01.337699 22456 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:01.343106 22456 tablet_bootstrap.cc:654] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.365658 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 8c934e1a62444e64911a279f6a544c96
I20250112 08:57:01.369000 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b61929ef8e8748c69fcf6bd20bf1c67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:01.371097 22565 raft_consensus.cc:2267] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.372314 22556 tablet_replica.cc:331] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:01.374451 22458 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:01.374867 22458 ts_tablet_manager.cc:1397] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.005s
I20250112 08:57:01.377103 22458 raft_consensus.cc:357] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.377605 22557 ts_tablet_manager.cc:1918] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.377763 22458 raft_consensus.cc:383] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.378096 22557 log.cc:1198] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:01.378134 22458 raft_consensus.cc:738] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.378803 22557 ts_tablet_manager.cc:1939] T 35f60d1f4bed4721b73deb9ba787417c P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:01.378782 22458 consensus_queue.cc:260] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.384578 22458 ts_tablet_manager.cc:1428] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20250112 08:57:01.385866 22458 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:01.385988 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_794e751834834ecebf5515268c2a7eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:01.386574 22710 raft_consensus.cc:491] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.387080 22710 raft_consensus.cc:513] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:01.387681 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_794e751834834ecebf5515268c2a7eb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.389406 22710 leader_election.cc:290] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.390702 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_9420682ee5cb40e893640510c8fafecb" } modify_external_catalogs: true
I20250112 08:57:01.391239 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_9420682ee5cb40e893640510c8fafecb" } modify_external_catalogs: true
I20250112 08:57:01.402278 22461 tablet_bootstrap.cc:654] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.402567 22458 tablet_bootstrap.cc:654] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.411571 22565 ts_tablet_manager.cc:1905] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.415328 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 93be4e0fc31c429f860c1a0cec64849b (table default.Table_8dd89a8dac4a4501a9cbe3108faab913 [id=2c86b027468242f694274397ec685267]) successfully deleted
I20250112 08:57:01.419668 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c934e1a62444e64911a279f6a544c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.420012 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c934e1a62444e64911a279f6a544c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.424604 22556 raft_consensus.cc:2238] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.426556 22556 raft_consensus.cc:2267] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.429179 22556 ts_tablet_manager.cc:1905] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:01.434185 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 105038us
I20250112 08:57:01.443635 22461 tablet_bootstrap.cc:492] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:01.444756 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:01.446856 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72d2062bf2a4c79b5a8be974d1b8cb5" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.447631 22211 raft_consensus.cc:2388] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 76718d1596134c83900541c6d398d59e in this term.
I20250112 08:57:01.448730 22812 raft_consensus.cc:491] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.449663 22458 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:01.450038 22458 ts_tablet_manager.cc:1397] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.003s
I20250112 08:57:01.449678 22812 raft_consensus.cc:513] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.451550 22812 leader_election.cc:290] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.452402 22461 ts_tablet_manager.cc:1397] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.006s
I20250112 08:57:01.452313 22458 raft_consensus.cc:357] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.452935 22458 raft_consensus.cc:383] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.453193 22458 raft_consensus.cc:738] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.453739 22458 consensus_queue.cc:260] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.455138 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c934e1a62444e64911a279f6a544c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.458989 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:01.459674 22068 raft_consensus.cc:2463] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:01.460912 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 106) took 1689 ms. Trace:
I20250112 08:57:01.461277 22557 rpcz_store.cc:276] 0112 08:56:59.771343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:56:59.771498 (+ 155us) service_pool.cc:225] Handling call
0112 08:57:01.192780 (+1421282us) ts_tablet_manager.cc:1145] Deleting tablet 35f60d1f4bed4721b73deb9ba787417c
0112 08:57:01.192822 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.316756 (+123934us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.377358 (+ 60602us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.460853 (+ 83495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420695}
I20250112 08:57:01.462121 22736 raft_consensus.cc:491] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.462574 22736 raft_consensus.cc:513] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.463665 22557 tablet_replica.cc:331] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:01.464439 22736 leader_election.cc:290] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
W20250112 08:57:01.472458 22290 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:01.473340 22813 consensus_queue.cc:1035] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:57:01.473773 22461 raft_consensus.cc:357] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.474413 22461 raft_consensus.cc:383] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.474675 22461 raft_consensus.cc:738] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.475221 22461 consensus_queue.cc:260] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.476625 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:01.477820 22146 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.478952 22716 consensus_queue.cc:1035] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:01.480346 22290 leader_election.cc:304] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:01.480782 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_7971251621ad4bb69351f4beb52fa190 [id=f69f93bdf75d4f1fb836d8961cbc68c4]) successfully deleted
I20250112 08:57:01.484210 22716 raft_consensus.cc:3054] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.436618 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet f252f0ec641d4bf5b4fe1399a8ced51e (table default.Table_4165821e9c5947b3bc177a3c85d8ff79 [id=902f4d86e8494b4aae4eb3ccf8690a9c]) successfully deleted
W20250112 08:57:01.487754 21704 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 51160.1us
W20250112 08:57:01.488408 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b61929ef8e8748c69fcf6bd20bf1c67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.490764 22716 raft_consensus.cc:2743] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:01.492923 22290 leader_election.cc:304] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:01.464452 22557 raft_consensus.cc:2238] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.496531 22557 raft_consensus.cc:2267] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.498142 22557 ts_tablet_manager.cc:1905] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.498471 22716 raft_consensus.cc:2798] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:01.499105 22716 raft_consensus.cc:491] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:01.499488 22716 raft_consensus.cc:3054] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.482586 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 35f60d1f4bed4721b73deb9ba787417c (table default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]) successfully deleted
W20250112 08:57:01.507957 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 2e62435dce1441a3b2fee50518662d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e62435dce1441a3b2fee50518662d48 already in progress: creating tablet
I20250112 08:57:01.512423 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1bca635d7c834c17893a9b550a3467d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.513895 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bca635d7c834c17893a9b550a3467d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.514405 22456 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:01.514791 22456 ts_tablet_manager.cc:1397] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.177s user 0.005s sys 0.016s
I20250112 08:57:01.516366 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b1c7befb37834969a004951fec5cc133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.517732 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1c7befb37834969a004951fec5cc133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.517457 22456 raft_consensus.cc:357] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.518090 22456 raft_consensus.cc:383] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.518327 22456 raft_consensus.cc:738] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.518905 22456 consensus_queue.cc:260] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.520332 21712 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_006c78dcc47545e1b7eb793c44633534" } new_table_name: "default.Table_a0ed97a52d8140f1869abb6a3095add6" modify_external_catalogs: true
I20250112 08:57:01.531004 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:01.532228 22149 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.532591 22149 leader_election.cc:304] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:01.533460 22736 raft_consensus.cc:2743] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:01.497649 22461 ts_tablet_manager.cc:1428] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20250112 08:57:01.539872 22461 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:01.557801 22557 ts_tablet_manager.cc:1918] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.560063 22557 log.cc:1198] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:01.560770 22557 ts_tablet_manager.cc:1939] T 06717cb567a54ee6a268c3be5d8786ec P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:01.563227 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 109) took 1055 ms. Trace:
I20250112 08:57:01.568578 22557 rpcz_store.cc:276] 0112 08:57:00.507867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:00.508018 (+ 151us) service_pool.cc:225] Handling call
0112 08:57:01.463526 (+955508us) ts_tablet_manager.cc:1145] Deleting tablet 06717cb567a54ee6a268c3be5d8786ec
0112 08:57:01.463560 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.545456 (+ 81896us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.557597 (+ 12141us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.563176 (+ 5579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955001}
I20250112 08:57:01.543222 22716 raft_consensus.cc:513] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.571007 22458 ts_tablet_manager.cc:1428] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.121s user 0.005s sys 0.001s
I20250112 08:57:01.572633 22716 leader_election.cc:290] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.574720 22456 ts_tablet_manager.cc:1428] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20250112 08:57:01.575779 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet 69651ab160db41b3a187542ac1cd2839 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:01.582868 22456 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:01.583969 22557 tablet_replica.cc:331] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:01.585261 22557 raft_consensus.cc:2238] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.586546 22557 raft_consensus.cc:2267] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:01.588778 22290 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:01.571856 22458 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:01.592226 22557 ts_tablet_manager.cc:1905] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.592660 22556 ts_tablet_manager.cc:1918] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:01.593004 22556 log.cc:1198] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:57:01.597461 22456 tablet_bootstrap.cc:654] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.574004 22064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabc37f08d1948118b98ba25b689bea4" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:01.609433 22556 ts_tablet_manager.cc:1939] T 6c683c8c64a441e8b5be52b077fdcb08 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:01.606154 22064 raft_consensus.cc:3054] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.616501 22812 raft_consensus.cc:491] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.625239 22812 raft_consensus.cc:513] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.627139 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_304a19a1f72e4fbd984a0b9156468aa0" } new_table_name: "default.Table_397822bc7cf943e7a8ab1fcd91cd6a31" modify_external_catalogs: true
I20250112 08:57:01.628217 22812 leader_election.cc:290] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.630561 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" } modify_external_catalogs: true
I20250112 08:57:01.631088 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" } modify_external_catalogs: true
I20250112 08:57:01.639506 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:01.641587 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 69651ab160db41b3a187542ac1cd2839 (table default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]). Expected version 1 got 0
I20250112 08:57:01.642166 21708 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:01.642127 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182470e64b14e329b278680dc132fcb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.643446 22211 raft_consensus.cc:2388] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 76718d1596134c83900541c6d398d59e in this term.
I20250112 08:57:01.647306 22557 ts_tablet_manager.cc:1918] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:01.647657 22557 log.cc:1198] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:01.649241 22565 ts_tablet_manager.cc:1918] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:01.649591 22565 log.cc:1198] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/6c683c8c64a441e8b5be52b077fdcb08
I20250112 08:57:01.649547 22706 raft_consensus.cc:491] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.650321 22565 ts_tablet_manager.cc:1939] T 6c683c8c64a441e8b5be52b077fdcb08 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:01.650004 22706 raft_consensus.cc:513] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.651355 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 6c683c8c64a441e8b5be52b077fdcb08 (table default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]) successfully deleted
I20250112 08:57:01.651760 22706 leader_election.cc:290] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.653486 22290 leader_election.cc:304] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:01.653599 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.654296 22211 raft_consensus.cc:2463] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:01.655591 22682 raft_consensus.cc:3054] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.655243 21999 leader_election.cc:304] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:01.656567 22706 raft_consensus.cc:2798] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:01.656847 22706 raft_consensus.cc:491] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:01.657121 22706 raft_consensus.cc:3054] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.663961 22682 raft_consensus.cc:2743] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:01.664458 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f46707de727a49108afb12ea2cf08aa0 consensus state has no leader
I20250112 08:57:01.666054 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:01.667399 22002 leader_election.cc:343] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.663168 22706 raft_consensus.cc:513] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.675717 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:57:01.677423 22002 leader_election.cc:343] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.677520 22458 tablet_bootstrap.cc:654] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.679769 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:01.680912 22211 raft_consensus.cc:3054] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:01.681675 22556 tablet_replica.cc:331] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:01.682390 22556 raft_consensus.cc:2238] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.684613 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad (DEFAULT_TABLE table=default.Table_ec36ed57a80f4350bda6894141741ba3 [id=6cb07113f32f4f039b2f200179a14796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.685525 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a22f87fbbf48e88dbd5af418ee94ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.686239 22556 raft_consensus.cc:2267] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.689736 22187 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.690788 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e34e1d70a4c63956a5be660e02b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.693315 22188 tablet_service.cc:1467] Processing CreateTablet for tablet d686b08555414954aba52ea077889660 (DEFAULT_TABLE table=default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.694319 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d686b08555414954aba52ea077889660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.694447 22461 tablet_bootstrap.cc:654] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.698096 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 69651ab160db41b3a187542ac1cd2839
I20250112 08:57:01.699360 21709 catalog_manager.cc:6596] ReplaceTablet: tablet 69651ab160db41b3a187542ac1cd2839 deleted and replaced by tablet 24e924f523524bd0a387597399e8470a
I20250112 08:57:01.699537 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 05401e13af414ce9bf8fdb6fce76e254 (DEFAULT_TABLE table=default.Table_38c3f75d2c224c5fb9b7d1223a62721d [id=58e7ecf6efb94961abc393c5a3578a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.700563 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05401e13af414ce9bf8fdb6fce76e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.705857 22186 tablet_service.cc:1514] Processing DeleteTablet for tablet 69651ab160db41b3a187542ac1cd2839 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e924f523524bd0a387597399e8470a) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.705106 21710 catalog_manager.cc:5526] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:01.706776 21710 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 3 got 0
I20250112 08:57:01.707154 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 1df6282224b849a488071323e2e69e1c (DEFAULT_TABLE table=default.Table_3bb2b43e854340fba2b4c4ad8a3da14a [id=eea0053edb8c4c8f8d0bd48fd076a0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.707504 21710 catalog_manager.cc:5526] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:01.708210 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df6282224b849a488071323e2e69e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.709060 21710 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:01.709636 21710 catalog_manager.cc:5526] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:01.710753 21710 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:01.711664 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet b88d2f7ac1d4471887c12e9e65e7a105
I20250112 08:57:01.712769 22456 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:01.713156 22456 ts_tablet_manager.cc:1397] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.003s
I20250112 08:57:01.714655 22189 tablet_service.cc:1467] Processing CreateTablet for tablet ef9c33e005b645e29464178c92a2466a (DEFAULT_TABLE table=default.Table_7199b4dc8d794b8a813e20147ba3f6d1 [id=25b738a4ae004486a24e323869783e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.715358 22565 tablet_replica.cc:331] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:01.715646 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9c33e005b645e29464178c92a2466a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.715927 22456 raft_consensus.cc:357] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:01.716398 22565 raft_consensus.cc:2238] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.716912 22565 raft_consensus.cc:2267] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.716804 22186 tablet_service.cc:1467] Processing CreateTablet for tablet f46707de727a49108afb12ea2cf08aa0 (DEFAULT_TABLE table=default.Table_397822bc7cf943e7a8ab1fcd91cd6a31 [id=644fe986709840f6aea68e8586428ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.716565 22456 raft_consensus.cc:383] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:01.721359 22456 raft_consensus.cc:738] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:01.721951 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46707de727a49108afb12ea2cf08aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.722008 22456 consensus_queue.cc:260] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:01.728837 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_538457c36d5643a2a4f3a40ebbbb25f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.730499 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_538457c36d5643a2a4f3a40ebbbb25f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.732470 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6ad283a4f2094ace8a0022618ac2bd5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.733860 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ad283a4f2094ace8a0022618ac2bd5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.736552 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.738256 22456 ts_tablet_manager.cc:1428] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.025s user 0.003s sys 0.002s
I20250112 08:57:01.738058 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da (DEFAULT_TABLE table=default.Table_83fb15dd26114a99b446cc4d91c8dd14 [id=0996fef5d9ad4192ace2ea4ae5d04450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.739025 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c34fc4b6aaf45fdae76cf153129f2da. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:01.739687 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 53167910ab0a4accb1d1615d537ba7ae consensus state has no leader
I20250112 08:57:01.740935 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 72e8f51e50994929b40a5e0679b07d8c (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.741776 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e8f51e50994929b40a5e0679b07d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.744971 22193 tablet_service.cc:1467] Processing CreateTablet for tablet fc82026fcd1541418176a015ea3253cf (DEFAULT_TABLE table=default.Table_dba2962989f24aec9af818bd34ea9122 [id=131b023798eb4e77a12306fb24329147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.745878 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc82026fcd1541418176a015ea3253cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.739033 22456 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:01.756573 22456 tablet_bootstrap.cc:654] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:01.777926 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0dc30e0598234d76a083f38a01cfd342" } new_table_name: "default.Table_8e7ddc505dc344c09a21b83bc0aeefce" modify_external_catalogs: true
I20250112 08:57:01.779174 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_2fac454051bb4b13979f5eb5c01aa277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.781415 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fac454051bb4b13979f5eb5c01aa277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:01.781180 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_03386b30353a4168b90717350c207283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.782840 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03386b30353a4168b90717350c207283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:01.795096 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 141664us
I20250112 08:57:01.822005 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad (DEFAULT_TABLE table=default.Table_ec36ed57a80f4350bda6894141741ba3 [id=6cb07113f32f4f039b2f200179a14796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.823016 22329 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.823037 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a22f87fbbf48e88dbd5af418ee94ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.832834 22329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e34e1d70a4c63956a5be660e02b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.840399 21713 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:01.850023 22556 ts_tablet_manager.cc:1905] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.857942 21712 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 24e924f523524bd0a387597399e8470a
I20250112 08:57:01.858184 22182 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:01.862807 22836 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 08:57:01.864486 22572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:01.866935 22736 raft_consensus.cc:491] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.870280 22736 raft_consensus.cc:513] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.872910 22565 ts_tablet_manager.cc:1905] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.872974 22736 leader_election.cc:290] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:01.876120 22572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:01.878217 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_372c2c8732cb429f9a5f0510ea2caca1" } new_table_name: "default.Table_71bfb1b1b306446fbe0386f94c93805f" modify_external_catalogs: true
I20250112 08:57:01.880183 22557 ts_tablet_manager.cc:1939] T 010ce6f55c444fe0852ad72cc7383cc1 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:01.883174 22572 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250112 08:57:01.791767 21704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.252:43161: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250112 08:57:01.897964 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da16d10565a74243a9ab8c5264bf7b1a has no consensus state
I20250112 08:57:01.902665 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 110) took 1341 ms. Trace:
I20250112 08:57:01.903079 22557 rpcz_store.cc:276] 0112 08:57:00.561653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:00.561808 (+ 155us) service_pool.cc:225] Handling call
0112 08:57:01.583813 (+1022005us) ts_tablet_manager.cc:1145] Deleting tablet 010ce6f55c444fe0852ad72cc7383cc1
0112 08:57:01.583851 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.600169 (+ 16318us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.647097 (+ 46928us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.902609 (+255512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021516}
I20250112 08:57:01.687013 22706 leader_election.cc:290] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.900120 22759 raft_consensus.cc:491] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.904510 22759 raft_consensus.cc:513] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.906313 22759 leader_election.cc:290] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:01.906239 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 010ce6f55c444fe0852ad72cc7383cc1 (table default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]) successfully deleted
I20250112 08:57:01.906504 22557 tablet_replica.cc:331] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:01.907132 22557 raft_consensus.cc:2238] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:01.907655 22557 raft_consensus.cc:2267] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:01.909312 22557 ts_tablet_manager.cc:1905] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:01.912092 22211 raft_consensus.cc:2463] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
W20250112 08:57:01.916044 21704 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.252:43161, user_credentials={real_user=slave}} blocked reactor thread for 124280us
I20250112 08:57:01.915325 22044 tablet_service.cc:1467] Processing CreateTablet for tablet ef9c33e005b645e29464178c92a2466a (DEFAULT_TABLE table=default.Table_7199b4dc8d794b8a813e20147ba3f6d1 [id=25b738a4ae004486a24e323869783e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.913209 22047 tablet_service.cc:1467] Processing CreateTablet for tablet 05401e13af414ce9bf8fdb6fce76e254 (DEFAULT_TABLE table=default.Table_38c3f75d2c224c5fb9b7d1223a62721d [id=58e7ecf6efb94961abc393c5a3578a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.920876 22047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05401e13af414ce9bf8fdb6fce76e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.914145 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet 69651ab160db41b3a187542ac1cd2839 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e924f523524bd0a387597399e8470a) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.921640 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 6c683c8c64a441e8b5be52b077fdcb08 (table default.Table_f7f03b3fa53f45c2a41783c3d17dd7b5 [id=ebbb22b0e9604ee4892888b35c8cc8ed]) successfully deleted
W20250112 08:57:01.922587 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 7a679595edde4a1f9b7f0916b2d760f2 because the tablet was not found. No further retry: Not found: Tablet not found: 7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:01.922894 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9c33e005b645e29464178c92a2466a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.914726 22048 tablet_service.cc:1467] Processing CreateTablet for tablet f46707de727a49108afb12ea2cf08aa0 (DEFAULT_TABLE table=default.Table_397822bc7cf943e7a8ab1fcd91cd6a31 [id=644fe986709840f6aea68e8586428ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.923293 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ec36ed57a80f4350bda6894141741ba3" } modify_external_catalogs: true
I20250112 08:57:01.923800 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ec36ed57a80f4350bda6894141741ba3" } modify_external_catalogs: true
I20250112 08:57:01.924106 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46707de727a49108afb12ea2cf08aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.927238 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da (DEFAULT_TABLE table=default.Table_83fb15dd26114a99b446cc4d91c8dd14 [id=0996fef5d9ad4192ace2ea4ae5d04450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.931819 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 72e8f51e50994929b40a5e0679b07d8c (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.943923 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 05401e13af414ce9bf8fdb6fce76e254 (DEFAULT_TABLE table=default.Table_38c3f75d2c224c5fb9b7d1223a62721d [id=58e7ecf6efb94961abc393c5a3578a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.944860 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05401e13af414ce9bf8fdb6fce76e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.915584 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:01.951663 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 69651ab160db41b3a187542ac1cd2839 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e924f523524bd0a387597399e8470a) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:01.952030 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad (DEFAULT_TABLE table=default.Table_ec36ed57a80f4350bda6894141741ba3 [id=6cb07113f32f4f039b2f200179a14796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.953637 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a22f87fbbf48e88dbd5af418ee94ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.955031 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 1df6282224b849a488071323e2e69e1c (DEFAULT_TABLE table=default.Table_3bb2b43e854340fba2b4c4ad8a3da14a [id=eea0053edb8c4c8f8d0bd48fd076a0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.956020 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:01.956653 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:01.958689 22146 leader_election.cc:343] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.958912 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:01.959767 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:01.960417 22211 raft_consensus.cc:2463] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:01.960668 22334 tablet_service.cc:1467] Processing CreateTablet for tablet d686b08555414954aba52ea077889660 (DEFAULT_TABLE table=default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.962071 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d686b08555414954aba52ea077889660. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:01.960259 22002 leader_election.cc:343] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:01.964781 22332 tablet_service.cc:1467] Processing CreateTablet for tablet f46707de727a49108afb12ea2cf08aa0 (DEFAULT_TABLE table=default.Table_397822bc7cf943e7a8ab1fcd91cd6a31 [id=644fe986709840f6aea68e8586428ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.965857 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f46707de727a49108afb12ea2cf08aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.913916 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 1df6282224b849a488071323e2e69e1c (DEFAULT_TABLE table=default.Table_3bb2b43e854340fba2b4c4ad8a3da14a [id=eea0053edb8c4c8f8d0bd48fd076a0c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.912606 22042 tablet_service.cc:1467] Processing CreateTablet for tablet d686b08555414954aba52ea077889660 (DEFAULT_TABLE table=default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.911772 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.968132 22837 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 08:57:01.942570 22759 raft_consensus.cc:491] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:01.979790 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_489166811cdb4e77859c967c8cba6ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:01.981659 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_489166811cdb4e77859c967c8cba6ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:01.982182 22826 log.cc:926] Time spent T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250112 08:57:01.983421 22037 tablet_service.cc:1467] Processing CreateTablet for tablet fc82026fcd1541418176a015ea3253cf (DEFAULT_TABLE table=default.Table_dba2962989f24aec9af818bd34ea9122 [id=131b023798eb4e77a12306fb24329147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.984189 22556 ts_tablet_manager.cc:1918] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:01.984609 22556 log.cc:1198] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/c08341b5ce6040d5be30f324d0373e38
I20250112 08:57:01.985414 22556 ts_tablet_manager.cc:1939] T c08341b5ce6040d5be30f324d0373e38 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:01.990267 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c34fc4b6aaf45fdae76cf153129f2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.991199 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3e34e1d70a4c63956a5be660e02b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.992132 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d686b08555414954aba52ea077889660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.992996 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df6282224b849a488071323e2e69e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:01.996527 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_47c9d8c18de54c368d05e5bde3538f56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:01.990913 22336 tablet_service.cc:1467] Processing CreateTablet for tablet ef9c33e005b645e29464178c92a2466a (DEFAULT_TABLE table=default.Table_7199b4dc8d794b8a813e20147ba3f6d1 [id=25b738a4ae004486a24e323869783e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:01.998165 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9c33e005b645e29464178c92a2466a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:01.998293 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47c9d8c18de54c368d05e5bde3538f56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.002084 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df6282224b849a488071323e2e69e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.011821 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da (DEFAULT_TABLE table=default.Table_83fb15dd26114a99b446cc4d91c8dd14 [id=0996fef5d9ad4192ace2ea4ae5d04450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.013021 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c34fc4b6aaf45fdae76cf153129f2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.014128 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.016635 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 72e8f51e50994929b40a5e0679b07d8c (DEFAULT_TABLE table=default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.017817 22331 tablet_service.cc:1467] Processing CreateTablet for tablet fc82026fcd1541418176a015ea3253cf (DEFAULT_TABLE table=default.Table_dba2962989f24aec9af818bd34ea9122 [id=131b023798eb4e77a12306fb24329147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.018880 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc82026fcd1541418176a015ea3253cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.020257 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet a090d258647b4dd7bd91cf64011735dd with delete_type TABLET_DATA_DELETED (replaced by tablet 267ab3e5955b4d3bbeca44baf021b328) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.021538 22328 tablet_service.cc:1514] Processing DeleteTablet for tablet dabc37f08d1948118b98ba25b689bea4 with delete_type TABLET_DATA_DELETED (replaced by tablet 98ddac23210248e3be1aba6591641b2d) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.023646 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e8f51e50994929b40a5e0679b07d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.023790 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e8f51e50994929b40a5e0679b07d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.025674 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_12e188e2f6d24ce68f14d301fdbccf00" } new_table_name: "default.Table_fea71badc3904a0ea177f78ebd702d15" modify_external_catalogs: true
I20250112 08:57:02.028743 22556 tablet_replica.cc:331] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:02.029399 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet c08341b5ce6040d5be30f324d0373e38 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]) successfully deleted
W20250112 08:57:02.022347 22149 leader_election.cc:343] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.037050 22149 leader_election.cc:304] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.040131 22736 raft_consensus.cc:2743] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:02.042207 22290 consensus_peers.cc:487] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a090d258647b4dd7bd91cf64011735dd. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:02.044440 22456 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:02.044883 22456 ts_tablet_manager.cc:1397] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.306s user 0.021s sys 0.028s
I20250112 08:57:02.047448 22456 raft_consensus.cc:357] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.055526 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f4e24291a17545d187eb8bfacf3c3f4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:02.057075 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4e24291a17545d187eb8bfacf3c3f4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.057025 22666 raft_consensus.cc:491] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.057548 22666 raft_consensus.cc:513] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:01.975692 22759 raft_consensus.cc:513] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.059841 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0159265c659d4a8981db80776c88f10b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:02.060978 21711 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_83fb15dd26114a99b446cc4d91c8dd14" } modify_external_catalogs: true
W20250112 08:57:02.061305 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0159265c659d4a8981db80776c88f10b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.061636 21711 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_83fb15dd26114a99b446cc4d91c8dd14" } modify_external_catalogs: true
I20250112 08:57:02.065372 22556 raft_consensus.cc:2238] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:02.066498 22556 raft_consensus.cc:2267] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:02.076045 22456 raft_consensus.cc:383] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.076719 22456 raft_consensus.cc:738] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:02.078751 22456 consensus_queue.cc:260] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.080297 22556 ts_tablet_manager.cc:1905] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:02.104324 22461 tablet_bootstrap.cc:492] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:02.104427 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0a79b2dc1b074c48bf8b7d84bda58ac6" } new_table_name: "default.Table_60490ab3bd724ea8bb2f2f573b901938" modify_external_catalogs: true
I20250112 08:57:02.104763 22461 ts_tablet_manager.cc:1397] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.565s user 0.028s sys 0.016s
I20250112 08:57:02.107472 22461 raft_consensus.cc:357] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.108136 22461 raft_consensus.cc:383] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.108410 22461 raft_consensus.cc:738] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:02.109404 22456 ts_tablet_manager.cc:1428] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.064s user 0.005s sys 0.001s
W20250112 08:57:02.123493 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98ddac23210248e3be1aba6591641b2d consensus state has no leader
I20250112 08:57:02.127811 22461 consensus_queue.cc:260] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.146947 22456 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:02.149508 22461 ts_tablet_manager.cc:1428] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20250112 08:57:02.153543 22461 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:02.164636 22456 tablet_bootstrap.cc:654] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:02.177153 22556 ts_tablet_manager.cc:1918] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:57:02.191331 22556 log.cc:1198] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/10bf912f189a42c19deb744c1c9f4daf
I20250112 08:57:02.192534 22556 ts_tablet_manager.cc:1939] T 10bf912f189a42c19deb744c1c9f4daf P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:01.997198 22557 ts_tablet_manager.cc:1918] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250112 08:57:02.196882 22557 log.cc:1198] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/10bf912f189a42c19deb744c1c9f4daf
I20250112 08:57:02.197906 22557 ts_tablet_manager.cc:1939] T 10bf912f189a42c19deb744c1c9f4daf P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:02.200513 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 117) took 1225 ms. Trace:
I20250112 08:57:02.202515 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 110) took 1116 ms. Trace:
I20250112 08:57:02.182088 22461 tablet_bootstrap.cc:654] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:02.210888 22556 rpcz_store.cc:276] 0112 08:57:01.085682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.085803 (+ 121us) service_pool.cc:225] Handling call
0112 08:57:02.028588 (+942785us) ts_tablet_manager.cc:1145] Deleting tablet 10bf912f189a42c19deb744c1c9f4daf
0112 08:57:02.028627 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.090117 (+ 61490us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.176943 (+ 86826us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.202460 (+ 25517us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34650,"tablet-delete.queue_time_us":899506}
I20250112 08:57:02.221098 22556 tablet_replica.cc:331] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:02.221882 22556 raft_consensus.cc:2238] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:02.222781 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_7971251621ad4bb69351f4beb52fa190 [id=f69f93bdf75d4f1fb836d8961cbc68c4]) successfully deleted
I20250112 08:57:02.224203 22556 raft_consensus.cc:2267] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:02.230669 22556 ts_tablet_manager.cc:1905] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:02.242326 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_12e188e2f6d24ce68f14d301fdbccf00 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 3 got 0
I20250112 08:57:02.243048 22557 rpcz_store.cc:276] 0112 08:57:00.975058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:00.975180 (+ 122us) service_pool.cc:225] Handling call
0112 08:57:01.906351 (+931171us) ts_tablet_manager.cc:1145] Deleting tablet 10bf912f189a42c19deb744c1c9f4daf
0112 08:57:01.906401 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.975339 (+ 68938us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:01.996994 (+ 21655us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.200425 (+203431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":886661}
I20250112 08:57:02.243692 22461 tablet_bootstrap.cc:492] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:02.244077 22461 ts_tablet_manager.cc:1397] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.091s user 0.012s sys 0.000s
I20250112 08:57:02.244143 22557 tablet_replica.cc:331] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:02.243005 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:02.237532 22736 raft_consensus.cc:491] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.245250 22736 raft_consensus.cc:513] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:02.246655 22461 raft_consensus.cc:357] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.247222 22461 raft_consensus.cc:383] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.247344 22736 leader_election.cc:290] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:02.247457 22461 raft_consensus.cc:738] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:02.249001 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:02.251179 22149 leader_election.cc:343] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.251566 22330 tablet_service.cc:1467] Processing CreateTablet for tablet fd228f7a432b436581368da7eab3ef16 (DEFAULT_TABLE table=default.Table_b61929ef8e8748c69fcf6bd20bf1c67d [id=960814982a0b4f6bbb3d2032137bdd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.252120 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]). Expected version 1 got 0
I20250112 08:57:02.252614 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd228f7a432b436581368da7eab3ef16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.239737 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 6368a80a9fe9457a803e68a2392b638d (DEFAULT_TABLE table=default.Table_1bca635d7c834c17893a9b550a3467d6 [id=47fdb7a81d5c4a1da01b0ca6330c30a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.256556 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6368a80a9fe9457a803e68a2392b638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.260175 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 10bf912f189a42c19deb744c1c9f4daf (table default.Table_7971251621ad4bb69351f4beb52fa190 [id=f69f93bdf75d4f1fb836d8961cbc68c4]) successfully deleted
W20250112 08:57:02.261251 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c1c634513e14e818601894834c41378 consensus state has no leader
I20250112 08:57:02.260983 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 6368a80a9fe9457a803e68a2392b638d (DEFAULT_TABLE table=default.Table_1bca635d7c834c17893a9b550a3467d6 [id=47fdb7a81d5c4a1da01b0ca6330c30a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.262204 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6368a80a9fe9457a803e68a2392b638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.265374 21713 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 3 got 0
I20250112 08:57:02.274222 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 92a22f87fbbf48e88dbd5af418ee94ad
I20250112 08:57:02.274122 22331 tablet_service.cc:1467] Processing CreateTablet for tablet 54b34b09f56d4417a3f321599170e5fd (DEFAULT_TABLE table=default.Table_794e751834834ecebf5515268c2a7eb1 [id=7838ea67766144e094e6392661d918bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.275182 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b34b09f56d4417a3f321599170e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:02.275049 21712 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 267ab3e5955b4d3bbeca44baf021b328 consensus state has no leader
I20250112 08:57:02.276155 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.287716 21711 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 6c34fc4b6aaf45fdae76cf153129f2da
I20250112 08:57:01.973598 22064 raft_consensus.cc:2463] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:02.266072 22736 raft_consensus.cc:491] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.295049 22736 raft_consensus.cc:513] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.284878 22184 tablet_service.cc:1467] Processing CreateTablet for tablet fd228f7a432b436581368da7eab3ef16 (DEFAULT_TABLE table=default.Table_b61929ef8e8748c69fcf6bd20bf1c67d [id=960814982a0b4f6bbb3d2032137bdd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.298024 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd228f7a432b436581368da7eab3ef16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.285596 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 54b34b09f56d4417a3f321599170e5fd (DEFAULT_TABLE table=default.Table_794e751834834ecebf5515268c2a7eb1 [id=7838ea67766144e094e6392661d918bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.302560 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b34b09f56d4417a3f321599170e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.285811 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:02.307298 22736 leader_election.cc:290] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:02.304237 22461 consensus_queue.cc:260] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.319700 22332 tablet_service.cc:1467] Processing CreateTablet for tablet da16d10565a74243a9ab8c5264bf7b1a (DEFAULT_TABLE table=default.Table_71bfb1b1b306446fbe0386f94c93805f [id=f9fef379ee924c1c8fdc18e5b98fcb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.320694 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da16d10565a74243a9ab8c5264bf7b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.333158 22191 tablet_service.cc:1467] Processing CreateTablet for tablet da16d10565a74243a9ab8c5264bf7b1a (DEFAULT_TABLE table=default.Table_71bfb1b1b306446fbe0386f94c93805f [id=f9fef379ee924c1c8fdc18e5b98fcb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.334122 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da16d10565a74243a9ab8c5264bf7b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.336459 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:02.336746 22185 tablet_service.cc:1467] Processing CreateTablet for tablet b62bae16ab43414c8d8be2766802d4c7 (DEFAULT_TABLE table=default.Table_6632e6bd7efc4fecb864bea93b604e87 [id=e35c684b420f477e8e8fb3b9f8f058f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.337749 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62bae16ab43414c8d8be2766802d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.321115 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.347234 22334 tablet_service.cc:1467] Processing CreateTablet for tablet b62bae16ab43414c8d8be2766802d4c7 (DEFAULT_TABLE table=default.Table_6632e6bd7efc4fecb864bea93b604e87 [id=e35c684b420f477e8e8fb3b9f8f058f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.348121 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:02.348327 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62bae16ab43414c8d8be2766802d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.357868 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 24e924f523524bd0a387597399e8470a (DEFAULT_TABLE table=default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.359547 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e924f523524bd0a387597399e8470a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.244855 22557 raft_consensus.cc:2238] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:02.360226 22557 raft_consensus.cc:2267] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:02.362349 22456 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:02.362746 22456 ts_tablet_manager.cc:1397] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.216s user 0.005s sys 0.031s
I20250112 08:57:02.366300 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:02.367723 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 24e924f523524bd0a387597399e8470a (DEFAULT_TABLE table=default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.369062 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e924f523524bd0a387597399e8470a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.374387 22330 tablet_service.cc:1467] Processing CreateTablet for tablet f9835288a7974e85821b095c03e91886 (DEFAULT_TABLE table=default.Table_b1c7befb37834969a004951fec5cc133 [id=27a78b113eef45bc906cee222fcb2c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.375468 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9835288a7974e85821b095c03e91886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.377974 22565 ts_tablet_manager.cc:1918] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:02.378295 22565 log.cc:1198] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:57:02.378969 22565 ts_tablet_manager.cc:1939] T b0192d66e64d4d39b4bcfd6565dff723 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:02.379679 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 108) took 1341 ms. Trace:
I20250112 08:57:02.265568 21999 leader_election.cc:304] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.397475 22461 ts_tablet_manager.cc:1428] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.153s user 0.006s sys 0.000s
I20250112 08:57:02.399194 22335 tablet_service.cc:1467] Processing CreateTablet for tablet e61e80e259db4645942141754c8491a3 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.400242 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61e80e259db4645942141754c8491a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.401101 22461 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:02.407613 22149 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.385300 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc82026fcd1541418176a015ea3253cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.415987 22759 leader_election.cc:290] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.409360 22666 leader_election.cc:290] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.425459 22188 tablet_service.cc:1467] Processing CreateTablet for tablet f9835288a7974e85821b095c03e91886 (DEFAULT_TABLE table=default.Table_b1c7befb37834969a004951fec5cc133 [id=27a78b113eef45bc906cee222fcb2c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.426859 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9835288a7974e85821b095c03e91886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.429396 22189 tablet_service.cc:1467] Processing CreateTablet for tablet e61e80e259db4645942141754c8491a3 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:02.430560 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61e80e259db4645942141754c8491a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:02.383257 22565 rpcz_store.cc:276] 0112 08:57:01.038472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.051993 (+ 13521us) service_pool.cc:225] Handling call
0112 08:57:01.715155 (+663162us) ts_tablet_manager.cc:1145] Deleting tablet b0192d66e64d4d39b4bcfd6565dff723
0112 08:57:01.715213 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:01.886342 (+171129us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.377716 (+491374us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.379627 (+ 1911us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":654838}
I20250112 08:57:02.455214 22565 tablet_replica.cc:331] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:02.455953 22565 raft_consensus.cc:2238] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:02.502770 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_22ffcdcebd8b439da7d3241c7ca67a97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:02.504530 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22ffcdcebd8b439da7d3241c7ca67a97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.513130 22557 ts_tablet_manager.cc:1905] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:02.532965 22456 raft_consensus.cc:357] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.536310 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f0dab317898745629f19ef4a1712abe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:02.537871 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0dab317898745629f19ef4a1712abe3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.539928 22456 raft_consensus.cc:383] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.540230 22456 raft_consensus.cc:738] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:02.540894 22456 consensus_queue.cc:260] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.546854 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_a0fa6d4216744e2786892721357a3278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:02.548444 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0fa6d4216744e2786892721357a3278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.556015 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_6632e6bd7efc4fecb864bea93b604e87" } new_table_name: "default.Table_362daa956768401d9c96b43a10dbc12e" modify_external_catalogs: true
I20250112 08:57:02.568308 22832 raft_consensus.cc:491] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.568742 22832 raft_consensus.cc:513] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.570672 22832 leader_election.cc:290] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.571043 22832 raft_consensus.cc:491] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.571378 22832 raft_consensus.cc:513] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.573302 22832 leader_election.cc:290] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.577426 22461 tablet_bootstrap.cc:654] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:02.578816 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_9adfc21f03654f96a0f9e342c9177c83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:02.579700 22556 ts_tablet_manager.cc:1918] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:02.580124 22556 log.cc:1198] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/b0192d66e64d4d39b4bcfd6565dff723
I20250112 08:57:02.580298 22456 ts_tablet_manager.cc:1428] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.217s user 0.007s sys 0.001s
W20250112 08:57:02.580511 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9adfc21f03654f96a0f9e342c9177c83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:02.581717 22556 ts_tablet_manager.cc:1939] T b0192d66e64d4d39b4bcfd6565dff723 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:02.583650 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 112) took 1475 ms. Trace:
I20250112 08:57:02.595930 22556 rpcz_store.cc:276] 0112 08:57:01.108627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.116794 (+ 8167us) service_pool.cc:225] Handling call
0112 08:57:02.220957 (+1104163us) ts_tablet_manager.cc:1145] Deleting tablet b0192d66e64d4d39b4bcfd6565dff723
0112 08:57:02.220996 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.283646 (+ 62650us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.579443 (+295797us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.583592 (+ 4149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47596,"tablet-delete.queue_time_us":1084108}
I20250112 08:57:02.597193 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 117) took 1438 ms. Trace:
I20250112 08:57:02.597482 22556 rpcz_store.cc:276] 0112 08:57:01.159121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.159483 (+ 362us) service_pool.cc:225] Handling call
0112 08:57:02.596842 (+1437359us) ts_tablet_manager.cc:1145] Deleting tablet a44cad3bd5a84e75a41733d8d2f345a8
0112 08:57:02.596880 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.597135 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24647,"tablet-delete.queue_time_us":1436859}
I20250112 08:57:02.598439 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 118) took 1364 ms. Trace:
I20250112 08:57:02.598722 22556 rpcz_store.cc:276] 0112 08:57:01.233579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.233703 (+ 124us) service_pool.cc:225] Handling call
0112 08:57:02.598231 (+1364528us) ts_tablet_manager.cc:1145] Deleting tablet 69a9c2b5a5d044c789a3eda07e8a312d
0112 08:57:02.598263 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.598398 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363656}
I20250112 08:57:02.599537 22556 tablet_replica.cc:331] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:02.600617 22456 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:02.606683 21710 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_dba2962989f24aec9af818bd34ea9122" } modify_external_catalogs: true
I20250112 08:57:02.606859 21709 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b1c7befb37834969a004951fec5cc133" } modify_external_catalogs: true
I20250112 08:57:02.607203 21710 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_dba2962989f24aec9af818bd34ea9122" } modify_external_catalogs: true
I20250112 08:57:02.607295 21709 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b1c7befb37834969a004951fec5cc133" } modify_external_catalogs: true
I20250112 08:57:02.609372 22841 raft_consensus.cc:491] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.609840 22841 raft_consensus.cc:513] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.611619 22841 leader_election.cc:290] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.614562 22736 raft_consensus.cc:491] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.614985 22736 raft_consensus.cc:513] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.617507 22556 raft_consensus.cc:2238] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:02.618083 22556 raft_consensus.cc:2267] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:02.618429 22736 leader_election.cc:290] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:02.624948 22565 raft_consensus.cc:2267] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:02.625162 22850 raft_consensus.cc:2798] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:02.625581 22850 raft_consensus.cc:695] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:02.627374 22565 ts_tablet_manager.cc:1905] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:02.627637 22556 ts_tablet_manager.cc:1905] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:02.628383 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8eaaa4847a874dd19d4bda6a4d4fd632" } new_table_name: "default.Table_95d4394b178c46188d58426a815a0964" modify_external_catalogs: true
W20250112 08:57:02.633877 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 301546us
I20250112 08:57:02.636032 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ce6f55c444fe0852ad72cc7383cc1" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:02.637044 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06717cb567a54ee6a268c3be5d8786ec" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:02.632778 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 657335us
I20250112 08:57:02.646831 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet b0192d66e64d4d39b4bcfd6565dff723 (table default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]) successfully deleted
I20250112 08:57:02.653100 22458 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:02.653343 22456 tablet_bootstrap.cc:654] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:02.653689 22458 ts_tablet_manager.cc:1397] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 1.082s user 0.018s sys 0.009s
I20250112 08:57:02.662259 22736 raft_consensus.cc:491] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:02.662786 22736 raft_consensus.cc:513] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.664997 21711 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b] alter complete (version 4)
W20250112 08:57:02.665421 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b62bae16ab43414c8d8be2766802d4c7 consensus state has no leader
I20250112 08:57:02.668387 22736 leader_election.cc:290] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:02.668952 22842 consensus_queue.cc:1035] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250112 08:57:02.670948 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:02.672770 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:02.673763 22063 raft_consensus.cc:2463] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:02.673691 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:02.675108 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:02.655714 21999 leader_election.cc:304] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:02.680495 22146 leader_election.cc:343] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.680905 22146 leader_election.cc:304] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.681737 22736 raft_consensus.cc:2743] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:02.675776 22850 consensus_queue.cc:237] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:02.685298 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 43723.2us
I20250112 08:57:02.685639 21716 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet d8509fdc7d6b467ab8d445f8a15fa014 (table default.Table_8e7ddc505dc344c09a21b83bc0aeefce [id=b9ecc50f7e5a454394c47cf7d253ec79]). Expected version 1 got 0
I20250112 08:57:02.687080 22146 leader_election.cc:304] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
W20250112 08:57:02.688076 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet a44cad3bd5a84e75a41733d8d2f345a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a44cad3bd5a84e75a41733d8d2f345a8 already in progress: creating tablet
W20250112 08:57:02.688740 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 69a9c2b5a5d044c789a3eda07e8a312d because the tablet was not found. No further retry: Not found: Tablet not found: 69a9c2b5a5d044c789a3eda07e8a312d
I20250112 08:57:02.692056 22666 raft_consensus.cc:2798] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:02.692375 22666 raft_consensus.cc:491] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:02.692699 22666 raft_consensus.cc:3054] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:02.695995 22146 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.696543 22146 leader_election.cc:304] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.697263 22835 raft_consensus.cc:2798] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:02.697567 22835 raft_consensus.cc:491] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:02.698284 22835 raft_consensus.cc:3054] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:02.704833 22835 raft_consensus.cc:513] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.707221 22736 raft_consensus.cc:2743] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:02.711453 22146 leader_election.cc:343] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.712574 22835 leader_election.cc:290] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:02.714882 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:02.719491 22002 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:02.720122 22002 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:02.720759 22002 leader_election.cc:343] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.737617 22759 consensus_queue.cc:1035] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:02.742355 22666 raft_consensus.cc:513] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:02.747644 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_362daa956768401d9c96b43a10dbc12e [id=e35c684b420f477e8e8fb3b9f8f058f5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b62bae16ab43414c8d8be2766802d4c7 consensus state has no leader
I20250112 08:57:02.748078 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:02.748818 22213 raft_consensus.cc:2388] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 53068c99960647728141706e9c50b870 in this term.
I20250112 08:57:02.744591 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:02.684209 22458 raft_consensus.cc:357] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.759722 22458 raft_consensus.cc:383] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.760038 22458 raft_consensus.cc:738] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
W20250112 08:57:02.762765 21999 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:02.763115 22666 leader_election.cc:290] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:02.764318 22062 raft_consensus.cc:3054] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:02.763252 21999 leader_election.cc:304] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:02.766319 21999 consensus_peers.cc:487] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:02.763607 22842 consensus_queue.cc:1035] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:02.754745 22556 ts_tablet_manager.cc:1918] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:02.768203 22556 log.cc:1198] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/7451e83f3b554936a79c6724ebe76cbb
I20250112 08:57:02.768886 22556 ts_tablet_manager.cc:1939] T 7451e83f3b554936a79c6724ebe76cbb P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:02.773742 22853 raft_consensus.cc:2743] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:02.778401 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 89562.1us
I20250112 08:57:02.781647 21710 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet fc82026fcd1541418176a015ea3253cf
I20250112 08:57:02.775993 22458 consensus_queue.cc:260] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.785931 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet f9835288a7974e85821b095c03e91886
I20250112 08:57:02.785964 22873 tablet.cc:1721] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 08:57:02.784392 22666 consensus_queue.cc:1035] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20250112 08:57:02.800618 22290 leader_election.cc:343] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 93be4e0fc31c429f860c1a0cec64849b
W20250112 08:57:02.801347 22290 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:02.808022 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
W20250112 08:57:02.811162 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ad9b1593a24424989f380811954439e consensus state has no leader
W20250112 08:57:02.817091 22002 consensus_peers.cc:487] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:02.817690 22002 consensus_peers.cc:487] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:02.821489 22461 tablet_bootstrap.cc:492] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:02.821936 22461 ts_tablet_manager.cc:1397] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.421s user 0.070s sys 0.046s
I20250112 08:57:02.824796 22461 raft_consensus.cc:357] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.825524 22461 raft_consensus.cc:383] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:02.825790 22461 raft_consensus.cc:738] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:02.826592 22461 consensus_queue.cc:260] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.829252 22461 ts_tablet_manager.cc:1428] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250112 08:57:02.830188 22461 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:02.835615 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:02.839257 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet f9835288a7974e85821b095c03e91886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:02.841091 22062 raft_consensus.cc:2463] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:02.841336 21999 leader_election.cc:304] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.852222 22146 leader_election.cc:304] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:02.854066 22853 raft_consensus.cc:2743] T da0c101de2544cc9907c78fdf1a4af82 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:02.855127 22736 raft_consensus.cc:2798] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 08:57:02.864102 21999 consensus_peers.cc:487] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93be4e0fc31c429f860c1a0cec64849b. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:02.864691 21999 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:02.865103 21999 leader_election.cc:304] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:02.866022 22666 raft_consensus.cc:2743] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:02.874608 22736 raft_consensus.cc:695] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:02.876214 22736 consensus_queue.cc:237] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:02.882136 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c7056aaf8844da915acec5f9f92eeb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:02.882742 22067 raft_consensus.cc:2463] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:02.905591 22028 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56470 (request call id 24) took 1065 ms. Trace:
I20250112 08:57:02.906154 22028 rpcz_store.cc:276] 0112 08:57:01.621321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.621461 (+ 140us) service_pool.cc:225] Handling call
0112 08:57:02.311204 (+689743us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54000a2b54. stack: 00007f8983942853 00007f8983942750 00007f89834b7584 00000000004fcd24 00000000004fccb9 00007f898a8426d1 00007f898a8b7268 00007f898b8ad6bd 00007f898a14be14 00007f898a14bda2 00007f898a14bcd1 00007f898a14bc4c 00007f898a14af02 00007f898868adfc 00007f898868a236 00007f8988689bdf
0112 08:57:02.686814 (+375610us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1524540160}
I20250112 08:57:02.915334 22043 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56484 (request call id 121) took 1006 ms. Trace:
I20250112 08:57:02.882093 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:02.948216 22212 raft_consensus.cc:3054] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:02.978750 22043 rpcz_store.cc:276] 0112 08:57:01.909035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.913330 (+ 4295us) service_pool.cc:225] Handling call
0112 08:57:01.966406 (+ 53076us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 08:57:01.966498 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0112 08:57:02.481474 (+514976us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.914806 (+433332us) tablet_replica.cc:206] Creating consensus instance
0112 08:57:02.915265 (+ 459us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 08:57:02.932166 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 114544us
I20250112 08:57:02.934343 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:02.935498 22038 tablet_service.cc:1514] Processing DeleteTablet for tablet dabc37f08d1948118b98ba25b689bea4 with delete_type TABLET_DATA_DELETED (replaced by tablet 98ddac23210248e3be1aba6591641b2d) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:02.937290 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93be4e0fc31c429f860c1a0cec64849b" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:02.980480 22002 leader_election.cc:343] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:02.940114 22041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56484 (request call id 118) took 1039 ms. Trace:
I20250112 08:57:02.986161 22041 rpcz_store.cc:276] 0112 08:57:01.908128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.911105 (+ 2977us) service_pool.cc:225] Handling call
0112 08:57:01.967356 (+ 56251us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 08:57:01.967431 (+ 75us) ts_tablet_manager.cc:676] Creating new metadata...
0112 08:57:02.490023 (+522592us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.939588 (+449565us) tablet_replica.cc:206] Creating consensus instance
0112 08:57:02.940052 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 08:57:02.941638 22458 ts_tablet_manager.cc:1428] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.288s user 0.006s sys 0.000s
I20250112 08:57:02.972165 22039 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:02.987977 22458 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:02.998924 22353 raft_consensus.cc:3054] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:03.005106 22565 ts_tablet_manager.cc:1918] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:03.005479 22565 log.cc:1198] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/7451e83f3b554936a79c6724ebe76cbb
I20250112 08:57:03.006174 22565 ts_tablet_manager.cc:1939] T 7451e83f3b554936a79c6724ebe76cbb P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:02.978839 22832 raft_consensus.cc:491] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:03.012267 22832 raft_consensus.cc:513] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.012785 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 6368a80a9fe9457a803e68a2392b638d (DEFAULT_TABLE table=default.Table_1bca635d7c834c17893a9b550a3467d6 [id=47fdb7a81d5c4a1da01b0ca6330c30a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.014230 22832 leader_election.cc:290] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:03.013897 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6368a80a9fe9457a803e68a2392b638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.024631 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 119) took 1771 ms. Trace:
I20250112 08:57:03.024976 22556 rpcz_store.cc:276] 0112 08:57:01.253065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.270355 (+ 17290us) service_pool.cc:225] Handling call
0112 08:57:02.599397 (+1329042us) ts_tablet_manager.cc:1145] Deleting tablet 7451e83f3b554936a79c6724ebe76cbb
0112 08:57:02.599436 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.646444 (+ 47008us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:02.754511 (+108067us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.024569 (+270058us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":209705,"tablet-delete.queue_time_us":1318765}
I20250112 08:57:03.025393 22041 tablet_service.cc:1467] Processing CreateTablet for tablet fd228f7a432b436581368da7eab3ef16 (DEFAULT_TABLE table=default.Table_b61929ef8e8748c69fcf6bd20bf1c67d [id=960814982a0b4f6bbb3d2032137bdd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.026084 22556 tablet_replica.cc:331] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:03.026464 22041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd228f7a432b436581368da7eab3ef16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.037433 22556 raft_consensus.cc:2238] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:03.038041 22556 raft_consensus.cc:2267] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:03.039855 22556 ts_tablet_manager.cc:1905] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:03.055853 22212 raft_consensus.cc:2463] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:03.057327 21999 leader_election.cc:304] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:03.059140 22853 raft_consensus.cc:2798] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:03.060101 22853 raft_consensus.cc:695] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:03.060911 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 54b34b09f56d4417a3f321599170e5fd (DEFAULT_TABLE table=default.Table_794e751834834ecebf5515268c2a7eb1 [id=7838ea67766144e094e6392661d918bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.062140 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54b34b09f56d4417a3f321599170e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.060899 22853 consensus_queue.cc:237] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.071192 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.075834 22037 tablet_service.cc:1467] Processing CreateTablet for tablet da16d10565a74243a9ab8c5264bf7b1a (DEFAULT_TABLE table=default.Table_71bfb1b1b306446fbe0386f94c93805f [id=f9fef379ee924c1c8fdc18e5b98fcb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.077952 22043 tablet_service.cc:1467] Processing CreateTablet for tablet b62bae16ab43414c8d8be2766802d4c7 (DEFAULT_TABLE table=default.Table_6632e6bd7efc4fecb864bea93b604e87 [id=e35c684b420f477e8e8fb3b9f8f058f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.084198 22666 consensus_queue.cc:1035] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250112 08:57:03.084896 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 244444us
I20250112 08:57:03.085884 21708 master_service.cc:946] ReplaceTablet: received request to replace tablet 06717cb567a54ee6a268c3be5d8786ec from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:03.088152 22331 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.088270 22042 tablet_service.cc:1467] Processing CreateTablet for tablet 24e924f523524bd0a387597399e8470a (DEFAULT_TABLE table=default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.088181 22335 tablet_service.cc:1514] Processing DeleteTablet for tablet f9835288a7974e85821b095c03e91886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.088414 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e72a7add315b4e81a640003870fc274a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:03.087509 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_bc2783a8f8df477e9555dfddfef9a722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:03.089104 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:03.089920 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72a7add315b4e81a640003870fc274a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:03.089980 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc2783a8f8df477e9555dfddfef9a722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:03.090044 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.091079 22048 tablet_service.cc:1467] Processing CreateTablet for tablet f9835288a7974e85821b095c03e91886 (DEFAULT_TABLE table=default.Table_b1c7befb37834969a004951fec5cc133 [id=27a78b113eef45bc906cee222fcb2c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.090844 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_222acc98e7144002a882838fb95ebf8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:03.092103 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c7056aaf8844da915acec5f9f92eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:55 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.092237 22044 tablet_service.cc:1467] Processing CreateTablet for tablet e61e80e259db4645942141754c8491a3 (DEFAULT_TABLE table=default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:03.092600 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_222acc98e7144002a882838fb95ebf8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:03.092942 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.093564 21710 master_service.cc:946] ReplaceTablet: received request to replace tablet 7451e83f3b554936a79c6724ebe76cbb from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:03.093631 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.094210 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet f9835288a7974e85821b095c03e91886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.094521 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_22ffcdcebd8b439da7d3241c7ca67a97" } new_table_name: "default.Table_2b22da2a0636476d9ba405517af8d41c" modify_external_catalogs: true
I20250112 08:57:03.094870 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:03.096125 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:03.098117 22706 consensus_queue.cc:1035] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:03.099437 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:03.101146 22189 consensus_meta.cc:287] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Time spent flushing consensus metadata: real 0.603s user 0.004s sys 0.004s
I20250112 08:57:03.104934 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da16d10565a74243a9ab8c5264bf7b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.109117 22461 tablet_bootstrap.cc:654] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:03.113492 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e924f523524bd0a387597399e8470a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.116314 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9835288a7974e85821b095c03e91886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.116863 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61e80e259db4645942141754c8491a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.145094 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
W20250112 08:57:03.147534 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4fc4d2d0eaa47f6a02f518828253057 has no consensus state
I20250112 08:57:03.156273 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 7451e83f3b554936a79c6724ebe76cbb (table default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]) successfully deleted
I20250112 08:57:03.157003 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b62bae16ab43414c8d8be2766802d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:03.160849 22149 leader_election.cc:343] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:03.162048 22149 leader_election.cc:343] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:03.165230 22149 leader_election.cc:343] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:03.165591 22149 leader_election.cc:304] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:03.167083 22887 raft_consensus.cc:2743] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:03.168211 22290 leader_election.cc:304] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:03.168972 22832 raft_consensus.cc:2798] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:03.169392 22832 raft_consensus.cc:695] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:03.170233 22832 consensus_queue.cc:237] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.177567 22212 raft_consensus.cc:1270] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Refusing update from remote peer 76718d1596134c83900541c6d398d59e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 08:57:03.183017 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 85080.3us
I20250112 08:57:03.183434 22706 consensus_queue.cc:1035] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:03.188028 22919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.252:43161 every 8 connection(s)
I20250112 08:57:03.189206 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 110) took 1950 ms. Trace:
I20250112 08:57:03.189561 22565 rpcz_store.cc:276] 0112 08:57:01.239170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.239301 (+ 131us) service_pool.cc:225] Handling call
0112 08:57:02.455058 (+1215757us) ts_tablet_manager.cc:1145] Deleting tablet 7451e83f3b554936a79c6724ebe76cbb
0112 08:57:02.455106 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.638512 (+183406us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.004878 (+366366us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.189144 (+184266us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":168506,"tablet-delete.queue_time_us":1215271}
I20250112 08:57:03.190672 22565 tablet_replica.cc:331] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:03.191869 22565 raft_consensus.cc:2238] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:03.192343 22565 raft_consensus.cc:2267] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:03.194020 22565 ts_tablet_manager.cc:1905] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:03.195093 22572 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.252:43161
I20250112 08:57:03.209779 22873 tablet.cc:1721] T d8509fdc7d6b467ab8d445f8a15fa014 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 08:57:03.215008 22911 log.cc:926] Time spent T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250112 08:57:03.217550 22458 tablet_bootstrap.cc:654] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:03.220419 22572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
I20250112 08:57:03.225831 22456 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:03.226377 22456 ts_tablet_manager.cc:1397] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.626s user 0.014s sys 0.012s
I20250112 08:57:03.227502 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 22572
I20250112 08:57:03.228803 22456 raft_consensus.cc:357] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.229403 22456 raft_consensus.cc:383] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:03.229619 22456 raft_consensus.cc:738] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:03.230553 22557 ts_tablet_manager.cc:1918] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:03.230227 22456 consensus_queue.cc:260] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.230868 22557 log.cc:1198] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/65591a0bb44148c49715b4564dd13524
I20250112 08:57:03.231513 22557 ts_tablet_manager.cc:1939] T 65591a0bb44148c49715b4564dd13524 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:57:03.233700 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 65555.6us
I20250112 08:57:03.234601 22859 consensus_queue.cc:1035] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:03.235855 22859 consensus_queue.cc:1035] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:03.252406 21713 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_97d2a91e67624174904c1f044679de1f" } modify_external_catalogs: true
I20250112 08:57:03.252990 21713 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_97d2a91e67624174904c1f044679de1f" } modify_external_catalogs: true
I20250112 08:57:03.252362 22923 tablet.cc:1721] T d8509fdc7d6b467ab8d445f8a15fa014 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:03.264528 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_de19595082ed4dc890bbceb724ea6b19" } modify_external_catalogs: true
I20250112 08:57:03.265277 22290 leader_election.cc:304] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:03.265316 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_de19595082ed4dc890bbceb724ea6b19" } modify_external_catalogs: true
I20250112 08:57:03.267256 22832 raft_consensus.cc:2798] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:03.267603 22832 raft_consensus.cc:491] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:03.268020 22832 raft_consensus.cc:3054] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:03.294426 21713 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:03.295167 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 253b5b6ea6f044fc8dcaafdc2e587f84
I20250112 08:57:03.315047 22461 tablet_bootstrap.cc:492] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:03.315575 22461 ts_tablet_manager.cc:1397] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.486s user 0.006s sys 0.003s
I20250112 08:57:03.321120 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 163d4e4a20cd4f0e80e559fdd04343b4 (DEFAULT_TABLE table=default.Table_489166811cdb4e77859c967c8cba6ed6 [id=0792e455dfe944d6a2fb33a01a524a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.322443 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163d4e4a20cd4f0e80e559fdd04343b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.324532 22456 ts_tablet_manager.cc:1428] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.098s user 0.005s sys 0.000s
I20250112 08:57:03.325524 22456 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:03.325875 22261 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
W20250112 08:57:03.334177 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 68976.1us
I20250112 08:57:03.334929 22290 leader_election.cc:304] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:03.335161 22928 sys_catalog.cc:263] Verifying existing consensus state
I20250112 08:57:03.337977 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 163d4e4a20cd4f0e80e559fdd04343b4 (DEFAULT_TABLE table=default.Table_489166811cdb4e77859c967c8cba6ed6 [id=0792e455dfe944d6a2fb33a01a524a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.338210 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 74437c3dea6547f3b9545da3f8f971bb (DEFAULT_TABLE table=default.Table_0159265c659d4a8981db80776c88f10b [id=3d1f3e4fa4004b17b07b0978a27aff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.339227 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74437c3dea6547f3b9545da3f8f971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.339553 22331 tablet_service.cc:1467] Processing CreateTablet for tablet 153bb666e5d5440ea22d59260a8299c6 (DEFAULT_TABLE table=default.Table_f4e24291a17545d187eb8bfacf3c3f4d [id=981ab568694b453baf8c75fc5df5875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.340590 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153bb666e5d5440ea22d59260a8299c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.340622 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.342482 22556 ts_tablet_manager.cc:1918] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:03.342532 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c (DEFAULT_TABLE table=default.Table_03386b30353a4168b90717350c207283 [id=1a5a5aa4678d4b51a4d6d949a3f0536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.342828 22556 log.cc:1198] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/8c934e1a62444e64911a279f6a544c96
I20250112 08:57:03.343490 22334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.348886 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163d4e4a20cd4f0e80e559fdd04343b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:03.353179 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 56207.8us
I20250112 08:57:03.368714 22936 tablet.cc:1721] T 6182470e64b14e329b278680dc132fcb P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:57:03.372565 22887 raft_consensus.cc:491] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:03.372980 22887 raft_consensus.cc:513] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.376168 22887 leader_election.cc:290] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:03.378122 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:03.379631 22146 leader_election.cc:343] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:03.379443 22832 raft_consensus.cc:513] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.380636 22937 tablet.cc:1721] T d8509fdc7d6b467ab8d445f8a15fa014 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:03.381532 22403 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
I20250112 08:57:03.385715 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Bootstrap starting.
I20250112 08:57:03.319273 22461 raft_consensus.cc:357] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.391255 22461 raft_consensus.cc:383] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:03.391542 22461 raft_consensus.cc:738] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:03.392159 22461 consensus_queue.cc:260] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.343529 22556 ts_tablet_manager.cc:1939] T 8c934e1a62444e64911a279f6a544c96 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:03.344663 22338 tablet_service.cc:1467] Processing CreateTablet for tablet deac41b2a0584528b048e2a0ef09485e (DEFAULT_TABLE table=default.Table_2fac454051bb4b13979f5eb5c01aa277 [id=daf01888e088422f94f57e834bb548b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.344789 22333 tablet_service.cc:1467] Processing CreateTablet for tablet c228f04faa5c4f68a95adfb295235047 (DEFAULT_TABLE table=default.Table_538457c36d5643a2a4f3a40ebbbb25f9 [id=929d59bdba3141f5947bc3af54185497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.345397 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.345794 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 340f93f630a34b3b9a9386999e2684e8 (DEFAULT_TABLE table=default.Table_47c9d8c18de54c368d05e5bde3538f56 [id=453899f3df2e41849bdc3cdd8c32b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.346853 22330 tablet_service.cc:1467] Processing CreateTablet for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 (DEFAULT_TABLE table=default.Table_6ad283a4f2094ace8a0022618ac2bd5b [id=d3dfc1c9cf0b4d52a2e152226ce5273e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.400089 22113 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
W20250112 08:57:03.404309 22290 leader_election.cc:343] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Illegal state: Tablet not RUNNING: STOPPED
I20250112 08:57:03.419512 22928 log.cc:826] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:03.426626 22835 raft_consensus.cc:491] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:03.427208 22835 raft_consensus.cc:513] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.429975 22835 leader_election.cc:290] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:03.432088 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:03.432926 22062 raft_consensus.cc:2463] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:03.434291 22146 leader_election.cc:304] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:03.435389 22887 raft_consensus.cc:2798] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:03.435760 22887 raft_consensus.cc:491] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:03.436125 22887 raft_consensus.cc:3054] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:03.381390 22832 leader_election.cc:290] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:03.474107 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 305aedf43e7241228ba88c90c329369d (DEFAULT_TABLE table=default.Table_222acc98e7144002a882838fb95ebf8e [id=82f928fe03234defb7dcd2a0e40b3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.477483 22873 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56484 (request call id 129) took 1531 ms. Trace:
I20250112 08:57:03.482898 22932 tablet.cc:1721] T 6182470e64b14e329b278680dc132fcb P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:57:03.484512 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=8 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250112 08:57:03.485704 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Bootstrap complete.
I20250112 08:57:03.492386 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65591a0bb44148c49715b4564dd13524" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:03.475399 22328 tablet_service.cc:1467] Processing CreateTablet for tablet 7aabb08ff90f42d186487dc5c99644d2 (DEFAULT_TABLE table=default.Table_a0fa6d4216744e2786892721357a3278 [id=a22d56347987446e84ea2a0dfc9a3ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.498837 22461 ts_tablet_manager.cc:1428] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.182s user 0.005s sys 0.001s
I20250112 08:57:03.516736 22327 tablet_service.cc:1467] Processing CreateTablet for tablet c49ec037c22a42108ecb48bc8270191f (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.521312 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 120) took 2066 ms. Trace:
I20250112 08:57:03.521770 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340f93f630a34b3b9a9386999e2684e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.521652 22556 rpcz_store.cc:276] 0112 08:57:01.454867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.455002 (+ 135us) service_pool.cc:225] Handling call
0112 08:57:03.025919 (+1570917us) ts_tablet_manager.cc:1145] Deleting tablet 8c934e1a62444e64911a279f6a544c96
0112 08:57:03.025982 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.115406 (+ 89424us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.342288 (+226882us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.521255 (+178967us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10572,"tablet-delete.queue_time_us":1570373}
I20250112 08:57:03.523165 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]) successfully deleted
I20250112 08:57:03.524412 22324 tablet_service.cc:1467] Processing CreateTablet for tablet 1869844a4d2b4a019dadb26d5908b520 (DEFAULT_TABLE table=default.Table_f0dab317898745629f19ef4a1712abe3 [id=482a04e7a8854fd3b42e28da88c8186f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.521337 22928 raft_consensus.cc:357] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:03.534415 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 120) took 2493 ms. Trace:
I20250112 08:57:03.534742 22557 rpcz_store.cc:276] 0112 08:57:01.040920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.041047 (+ 127us) service_pool.cc:225] Handling call
0112 08:57:02.244000 (+1202953us) ts_tablet_manager.cc:1145] Deleting tablet 65591a0bb44148c49715b4564dd13524
0112 08:57:02.244038 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:02.964940 (+720902us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.230336 (+265396us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.534349 (+304013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202380}
W20250112 08:57:03.542744 22290 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:03.545010 22850 consensus_queue.cc:1035] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:03.383615 22933 tablet.cc:1721] T 6182470e64b14e329b278680dc132fcb P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 08:57:03.553478 22928 raft_consensus.cc:738] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa33dde8b5248e7ab4d6087d8e9d801, State: Initialized, Role: FOLLOWER
I20250112 08:57:03.554847 22928 consensus_queue.cc:260] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:03.559656 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 74437c3dea6547f3b9545da3f8f971bb (DEFAULT_TABLE table=default.Table_0159265c659d4a8981db80776c88f10b [id=3d1f3e4fa4004b17b07b0978a27aff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.561005 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74437c3dea6547f3b9545da3f8f971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.569931 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bac902842453ba9d4ede5e8688f76 (DEFAULT_TABLE table=default.Table_9adfc21f03654f96a0f9e342c9177c83 [id=dd65bb8025a84452bd06a93435f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.571640 22326 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 (DEFAULT_TABLE table=default.Table_2b22da2a0636476d9ba405517af8d41c [id=3357a5e7910d4489a32169e40b98a5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.573482 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 153bb666e5d5440ea22d59260a8299c6 (DEFAULT_TABLE table=default.Table_f4e24291a17545d187eb8bfacf3c3f4d [id=981ab568694b453baf8c75fc5df5875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.574378 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153bb666e5d5440ea22d59260a8299c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:03.586697 22002 consensus_peers.cc:487] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c934e1a62444e64911a279f6a544c96. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:03.588018 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:03.589983 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c228f04faa5c4f68a95adfb295235047. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:03.592268 22149 leader_election.cc:343] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:03.617975 22456 tablet_bootstrap.cc:654] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:03.521040 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:03.639174 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet b0192d66e64d4d39b4bcfd6565dff723 (table default.Table_4baf4c4da83c4d1a91eba0ed4aa07b54 [id=e5097fef487042d2a33e57b5bc04aa52]) successfully deleted
I20250112 08:57:03.640996 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c7056aaf8844da915acec5f9f92eeb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:03.641556 22067 raft_consensus.cc:3054] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:03.648644 22067 raft_consensus.cc:2463] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:03.653769 22190 tablet_service.cc:1467] Processing CreateTablet for tablet c228f04faa5c4f68a95adfb295235047 (DEFAULT_TABLE table=default.Table_538457c36d5643a2a4f3a40ebbbb25f9 [id=929d59bdba3141f5947bc3af54185497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.654429 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c (DEFAULT_TABLE table=default.Table_03386b30353a4168b90717350c207283 [id=1a5a5aa4678d4b51a4d6d949a3f0536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.654731 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c228f04faa5c4f68a95adfb295235047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.655347 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.656602 22186 tablet_service.cc:1467] Processing CreateTablet for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 (DEFAULT_TABLE table=default.Table_6ad283a4f2094ace8a0022618ac2bd5b [id=d3dfc1c9cf0b4d52a2e152226ce5273e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.657536 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.660581 22185 tablet_service.cc:1467] Processing CreateTablet for tablet deac41b2a0584528b048e2a0ef09485e (DEFAULT_TABLE table=default.Table_2fac454051bb4b13979f5eb5c01aa277 [id=daf01888e088422f94f57e834bb548b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.661592 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deac41b2a0584528b048e2a0ef09485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.664376 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 340f93f630a34b3b9a9386999e2684e8 (DEFAULT_TABLE table=default.Table_47c9d8c18de54c368d05e5bde3538f56 [id=453899f3df2e41849bdc3cdd8c32b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.667351 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340f93f630a34b3b9a9386999e2684e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.668491 22290 leader_election.cc:304] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
W20250112 08:57:03.670392 22290 consensus_peers.cc:487] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:03.671144 22290 leader_election.cc:304] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:03.677699 22458 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:03.678164 22458 ts_tablet_manager.cc:1397] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.690s user 0.009s sys 0.002s
I20250112 08:57:03.680395 22458 raft_consensus.cc:357] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.695883 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 119) took 2655 ms. Trace:
I20250112 08:57:03.701905 22458 raft_consensus.cc:383] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:03.702245 22458 raft_consensus.cc:738] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:03.702960 22458 consensus_queue.cc:260] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.705480 22458 ts_tablet_manager.cc:1428] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20250112 08:57:03.716918 22458 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:03.696251 22557 rpcz_store.cc:276] 0112 08:57:01.040609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.041861 (+ 1252us) service_pool.cc:225] Handling call
0112 08:57:03.695633 (+2653772us) ts_tablet_manager.cc:1145] Deleting tablet 6ffbab945abd4fa58259a65e4fadd622
0112 08:57:03.695676 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.695827 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653245}
I20250112 08:57:03.730361 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 118) took 2690 ms. Trace:
I20250112 08:57:03.730757 22557 rpcz_store.cc:276] 0112 08:57:01.040307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.092000 (+ 51693us) service_pool.cc:225] Handling call
0112 08:57:03.730045 (+2638045us) ts_tablet_manager.cc:1145] Deleting tablet aa641fe365844c9eaa6ffcbf548d82f7
0112 08:57:03.730098 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.730289 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633403}
I20250112 08:57:03.732404 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 121) took 2471 ms. Trace:
I20250112 08:57:03.732682 22557 rpcz_store.cc:276] 0112 08:57:01.260565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.260722 (+ 157us) service_pool.cc:225] Handling call
0112 08:57:03.732101 (+2471379us) ts_tablet_manager.cc:1145] Deleting tablet a44cad3bd5a84e75a41733d8d2f345a8
0112 08:57:03.732136 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.732359 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470823}
I20250112 08:57:03.652909 21713 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 3 got 0
I20250112 08:57:03.753129 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet d8509fdc7d6b467ab8d445f8a15fa014 (table default.Table_8e7ddc505dc344c09a21b83bc0aeefce [id=b9ecc50f7e5a454394c47cf7d253ec79]). Expected version 1 got 0
I20250112 08:57:03.753918 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 6182470e64b14e329b278680dc132fcb (table default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b]). Expected version 3 got 0
I20250112 08:57:03.754536 21712 catalog_manager.cc:5526] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:03.756624 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:03.757706 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:03.758378 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:03.779340 22565 ts_tablet_manager.cc:1918] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 08:57:03.786132 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c4fc4d2d0eaa47f6a02f518828253057 (table default.Table_2b22da2a0636476d9ba405517af8d41c [id=3357a5e7910d4489a32169e40b98a5c5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c4fc4d2d0eaa47f6a02f518828253057 consensus state has no leader
I20250112 08:57:03.562309 22325 tablet_service.cc:1467] Processing CreateTablet for tablet 56b9c882084142118fe251e6540fd8be (DEFAULT_TABLE table=default.Table_bc2783a8f8df477e9555dfddfef9a722 [id=cde1ceb16c4741b7abdd6f2362454467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.792130 22927 raft_consensus.cc:2743] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:03.791778 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:57:03.792726 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:03.793210 22927 raft_consensus.cc:491] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:03.793604 22927 raft_consensus.cc:513] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.795357 22927 leader_election.cc:290] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:03.799314 22458 tablet_bootstrap.cc:654] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:03.811234 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 140147us
I20250112 08:57:03.812055 22928 sys_catalog.cc:564] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 08:57:03.812057 22149 leader_election.cc:343] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:03.812450 22149 leader_election.cc:304] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:03.779749 22565 log.cc:1198] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/a090d258647b4dd7bd91cf64011735dd
I20250112 08:57:03.814520 22941 raft_consensus.cc:2798] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:03.814783 22565 ts_tablet_manager.cc:1939] T a090d258647b4dd7bd91cf64011735dd P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:03.814966 22941 raft_consensus.cc:695] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:03.816725 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 111) took 2572 ms. Trace:
I20250112 08:57:03.815718 22941 consensus_queue.cc:237] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.817026 22565 rpcz_store.cc:276] 0112 08:57:01.244455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.244594 (+ 139us) service_pool.cc:225] Handling call
0112 08:57:03.190513 (+1945919us) ts_tablet_manager.cc:1145] Deleting tablet a090d258647b4dd7bd91cf64011735dd
0112 08:57:03.190556 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.688672 (+498116us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.779077 (+ 90405us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.816647 (+ 37570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1944994}
I20250112 08:57:03.818210 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 112) took 2571 ms. Trace:
I20250112 08:57:03.818490 22565 rpcz_store.cc:276] 0112 08:57:01.246559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.246669 (+ 110us) service_pool.cc:225] Handling call
0112 08:57:03.817974 (+2571305us) ts_tablet_manager.cc:1145] Deleting tablet 7a679595edde4a1f9b7f0916b2d760f2
0112 08:57:03.818011 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.818161 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2564869}
I20250112 08:57:03.819568 22565 tablet_replica.cc:331] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:03.821125 22876 raft_consensus.cc:1212] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.224] Dedup: 1.10->[1.11-1.224]
I20250112 08:57:03.827772 22329 tablet_service.cc:1514] Processing DeleteTablet for tablet 93be4e0fc31c429f860c1a0cec64849b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:03.830070 22328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aabb08ff90f42d186487dc5c99644d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.832808 22461 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:03.834604 22556 tablet_replica.cc:331] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:03.835299 22556 raft_consensus.cc:2238] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:03.835813 22556 pending_rounds.cc:62] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:03.836045 22556 pending_rounds.cc:69] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113409426621468672 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "69651ab160db41b3a187542ac1cd2839" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" dest_uuid: "8c608785d6874abf9cc28db11a979ddd" new_extra_config { } }
W20250112 08:57:03.837119 22556 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44720: Aborted: Op aborted
I20250112 08:57:03.838223 22556 raft_consensus.cc:2267] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:03.845679 22556 ts_tablet_manager.cc:1905] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:03.849985 22461 tablet_bootstrap.cc:654] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:03.851644 22565 raft_consensus.cc:2238] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:03.852463 22565 raft_consensus.cc:2267] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:03.856621 22458 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:03.856979 22458 ts_tablet_manager.cc:1397] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.140s user 0.005s sys 0.004s
I20250112 08:57:03.866263 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.866937 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305aedf43e7241228ba88c90c329369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.867667 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bac902842453ba9d4ede5e8688f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.867987 22458 raft_consensus.cc:357] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.868649 22458 raft_consensus.cc:383] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:03.868963 22327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49ec037c22a42108ecb48bc8270191f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.868896 22458 raft_consensus.cc:738] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:03.870268 22326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4fc4d2d0eaa47f6a02f518828253057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.870954 22565 ts_tablet_manager.cc:1905] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:03.872305 22458 consensus_queue.cc:260] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.876250 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deac41b2a0584528b048e2a0ef09485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.877272 22458 ts_tablet_manager.cc:1428] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250112 08:57:03.895066 22565 ts_tablet_manager.cc:1918] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:03.895349 22565 log.cc:1198] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/c08341b5ce6040d5be30f324d0373e38
I20250112 08:57:03.896986 22325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9c882084142118fe251e6540fd8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.699533 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 74437c3dea6547f3b9545da3f8f971bb (DEFAULT_TABLE table=default.Table_0159265c659d4a8981db80776c88f10b [id=3d1f3e4fa4004b17b07b0978a27aff3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.477945 22873 rpcz_store.cc:276] 0112 08:57:01.963011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.965720 (+ 2709us) service_pool.cc:225] Handling call
0112 08:57:03.477398 (+1511678us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:02.221917 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:57:02.385241 (+163324us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:57:02.387506 (+ 2265us) alter_schema_op.cc:66] Acquired schema lock
0112 08:57:02.407936 (+ 20430us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:57:02.408075 (+ 139us) alter_schema_op.cc:128] START. Timestamp: 1736672222407083
0112 08:57:02.408095 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 08:57:02.435494 (+ 27399us) log.cc:844] Serialized 173 byte log entry
0112 08:57:02.730856 (+295362us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b54000ce7a0. stack: 00007f8983942853 00007f8983942750 00007f89834b7584 00000000004fcd24 00000000004fccb9 00007f898a8426d1 00007f898a856676 00007f898a85000a 00007f898a84f274 00007f898a8377a5 00007f898a83c9d1 00007f898a83c959 00007f898a83c8e9 00007f898a83c8b1 00007f898a83ba7d 000000000059ea04
0112 08:57:03.209458 (+478602us) op_driver.cc:464] REPLICATION: finished
0112 08:57:03.209699 (+ 241us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:57:03.210982 (+ 1283us) log.cc:844] Serialized 5 byte log entry
0112 08:57:03.251744 (+ 40762us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.336200 (+ 84456us) log.cc:844] Serialized 14 byte log entry
0112 08:57:03.477198 (+140998us) alter_schema_op.cc:73] Released schema lock
0112 08:57:03.477244 (+ 46us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":95,"mutex_wait_us":3438,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":12658,"prepare.run_wall_time_us":461561,"raft.queue_time_us":30476,"raft.run_cpu_time_us":4161,"raft.run_wall_time_us":140519,"replication_time_us":801258,"spinlock_wait_cycles":16811008,"thread_start_us":196887,"threads_started":1,"wal-append.queue_time_us":185855}]]}
I20250112 08:57:03.697659 22042 tablet_service.cc:1467] Processing CreateTablet for tablet 163d4e4a20cd4f0e80e559fdd04343b4 (DEFAULT_TABLE table=default.Table_489166811cdb4e77859c967c8cba6ed6 [id=0792e455dfe944d6a2fb33a01a524a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.901316 22042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163d4e4a20cd4f0e80e559fdd04343b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.906148 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1869844a4d2b4a019dadb26d5908b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.734371 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 153bb666e5d5440ea22d59260a8299c6 (DEFAULT_TABLE table=default.Table_f4e24291a17545d187eb8bfacf3c3f4d [id=981ab568694b453baf8c75fc5df5875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.735713 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c (DEFAULT_TABLE table=default.Table_03386b30353a4168b90717350c207283 [id=1a5a5aa4678d4b51a4d6d949a3f0536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.735972 22048 tablet_service.cc:1467] Processing CreateTablet for tablet c228f04faa5c4f68a95adfb295235047 (DEFAULT_TABLE table=default.Table_538457c36d5643a2a4f3a40ebbbb25f9 [id=929d59bdba3141f5947bc3af54185497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.932921 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c228f04faa5c4f68a95adfb295235047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.933459 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153bb666e5d5440ea22d59260a8299c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.934082 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3fbe9ad4224e5e8904c1b7dc7e231c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.737380 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 340f93f630a34b3b9a9386999e2684e8 (DEFAULT_TABLE table=default.Table_47c9d8c18de54c368d05e5bde3538f56 [id=453899f3df2e41849bdc3cdd8c32b93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.944806 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340f93f630a34b3b9a9386999e2684e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.948221 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74437c3dea6547f3b9545da3f8f971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.738869 22040 tablet_service.cc:1467] Processing CreateTablet for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 (DEFAULT_TABLE table=default.Table_6ad283a4f2094ace8a0022618ac2bd5b [id=d3dfc1c9cf0b4d52a2e152226ce5273e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.949170 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:03.950819 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 138812us
I20250112 08:57:03.740255 22045 tablet_service.cc:1467] Processing CreateTablet for tablet deac41b2a0584528b048e2a0ef09485e (DEFAULT_TABLE table=default.Table_2fac454051bb4b13979f5eb5c01aa277 [id=daf01888e088422f94f57e834bb548b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:03.953243 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deac41b2a0584528b048e2a0ef09485e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:03.960165 22557 tablet_replica.cc:331] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:03.971395 22666 raft_consensus.cc:491] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:03.971853 22666 raft_consensus.cc:513] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:03.973686 22666 leader_election.cc:290] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:03.953310 22835 raft_consensus.cc:2743] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:03.987589 22557 raft_consensus.cc:2238] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 08:57:03.990533 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 351357us
I20250112 08:57:03.992210 22557 raft_consensus.cc:2267] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:03.992913 22887 consensus_meta.cc:287] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Time spent flushing consensus metadata: real 0.556s user 0.005s sys 0.005s
W20250112 08:57:04.034694 21999 consensus_peers.cc:487] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:04.035095 22458 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:04.037863 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:04.041720 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:04.042363 22212 raft_consensus.cc:2463] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:04.047782 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 7451e83f3b554936a79c6724ebe76cbb (table default.Table_8e031f733adc400f80ef5a314e2c2704 [id=da56c81e84f4409cb0786ddd4aceaee3]) successfully deleted
I20250112 08:57:04.050663 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet a090d258647b4dd7bd91cf64011735dd (table default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
W20250112 08:57:04.051496 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 7a679595edde4a1f9b7f0916b2d760f2 because the tablet was not found. No further retry: Not found: Tablet not found: 7a679595edde4a1f9b7f0916b2d760f2
I20250112 08:57:04.036031 22887 raft_consensus.cc:513] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.066213 22887 leader_election.cc:290] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:04.076720 22565 ts_tablet_manager.cc:1939] T c08341b5ce6040d5be30f324d0373e38 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:04.077033 22556 ts_tablet_manager.cc:1918] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:04.077390 22556 log.cc:1198] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/69651ab160db41b3a187542ac1cd2839
I20250112 08:57:04.078068 22556 ts_tablet_manager.cc:1939] T 69651ab160db41b3a187542ac1cd2839 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:04.079883 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 126) took 2129 ms. Trace:
I20250112 08:57:04.082065 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:04.082602 22068 raft_consensus.cc:3054] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.083496 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0c101de2544cc9907c78fdf1a4af82" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:04.085844 22557 ts_tablet_manager.cc:1905] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:04.080243 22556 rpcz_store.cc:276] 0112 08:57:01.950736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.950862 (+ 126us) service_pool.cc:225] Handling call
0112 08:57:03.522780 (+1571918us) ts_tablet_manager.cc:1145] Deleting tablet 69651ab160db41b3a187542ac1cd2839
0112 08:57:03.834492 (+311712us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.926387 (+ 91895us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.076822 (+150435us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.079830 (+ 3008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1570648}
I20250112 08:57:04.087076 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 129) took 2076 ms. Trace:
I20250112 08:57:04.087380 22556 rpcz_store.cc:276] 0112 08:57:02.010653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.014003 (+ 3350us) service_pool.cc:225] Handling call
0112 08:57:04.086760 (+2072757us) ts_tablet_manager.cc:1145] Deleting tablet b88d2f7ac1d4471887c12e9e65e7a105
0112 08:57:04.086796 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.087024 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2072228}
I20250112 08:57:04.088297 22556 tablet_replica.cc:331] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
W20250112 08:57:04.045013 22358 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.000s sys 0.006s
I20250112 08:57:04.099421 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:04.104067 21999 leader_election.cc:304] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:04.105211 22666 raft_consensus.cc:2798] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:04.105544 22666 raft_consensus.cc:491] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:04.105859 22666 raft_consensus.cc:3054] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.110505 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:04.111124 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:04.111645 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:04.114881 22835 consensus_queue.cc:1035] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:04.115574 22068 raft_consensus.cc:2463] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
W20250112 08:57:04.116158 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet b88d2f7ac1d4471887c12e9e65e7a105 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b88d2f7ac1d4471887c12e9e65e7a105 already in progress: creating tablet
I20250112 08:57:04.116729 22146 leader_election.cc:304] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:04.117537 22887 raft_consensus.cc:2798] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:04.117980 22887 raft_consensus.cc:695] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
W20250112 08:57:04.108628 22358 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s user 0.000s sys 0.006s
I20250112 08:57:04.118798 22887 consensus_queue.cc:237] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.123395 21713 catalog_manager.cc:5526] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:04.126394 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:04.127928 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 65591a0bb44148c49715b4564dd13524 (table default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]) successfully deleted
W20250112 08:57:04.132122 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 6ffbab945abd4fa58259a65e4fadd622 because the tablet was not found. No further retry: Not found: Tablet not found: 6ffbab945abd4fa58259a65e4fadd622
W20250112 08:57:04.137091 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet aa641fe365844c9eaa6ffcbf548d82f7 because the tablet was not found. No further retry: Not found: Tablet not found: aa641fe365844c9eaa6ffcbf548d82f7
I20250112 08:57:04.150535 22931 raft_consensus.cc:2743] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:04.156100 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet a44cad3bd5a84e75a41733d8d2f345a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a44cad3bd5a84e75a41733d8d2f345a8 already in progress: creating tablet
I20250112 08:57:04.162786 22355 raft_consensus.cc:3054] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.172283 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:04.201081 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 305aedf43e7241228ba88c90c329369d (DEFAULT_TABLE table=default.Table_222acc98e7144002a882838fb95ebf8e [id=82f928fe03234defb7dcd2a0e40b3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.202222 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 7aabb08ff90f42d186487dc5c99644d2 (DEFAULT_TABLE table=default.Table_a0fa6d4216744e2786892721357a3278 [id=a22d56347987446e84ea2a0dfc9a3ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.204690 22187 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 (DEFAULT_TABLE table=default.Table_2b22da2a0636476d9ba405517af8d41c [id=3357a5e7910d4489a32169e40b98a5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.206288 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 56b9c882084142118fe251e6540fd8be (DEFAULT_TABLE table=default.Table_bc2783a8f8df477e9555dfddfef9a722 [id=cde1ceb16c4741b7abdd6f2362454467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.207937 22182 tablet_service.cc:1467] Processing CreateTablet for tablet 1869844a4d2b4a019dadb26d5908b520 (DEFAULT_TABLE table=default.Table_f0dab317898745629f19ef4a1712abe3 [id=482a04e7a8854fd3b42e28da88c8186f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.203660 22192 tablet_service.cc:1467] Processing CreateTablet for tablet c49ec037c22a42108ecb48bc8270191f (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:04.212469 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 89041.8us
I20250112 08:57:04.212785 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:04.216851 22290 consensus_peers.cc:487] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dabc37f08d1948118b98ba25b689bea4. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:04.202256 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305aedf43e7241228ba88c90c329369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.219128 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aabb08ff90f42d186487dc5c99644d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.222550 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49ec037c22a42108ecb48bc8270191f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.224275 22182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1869844a4d2b4a019dadb26d5908b520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.228344 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9c882084142118fe251e6540fd8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.232252 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4fc4d2d0eaa47f6a02f518828253057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.213085 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:04.237730 22461 tablet_bootstrap.cc:492] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:04.238126 22461 ts_tablet_manager.cc:1397] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.406s user 0.020s sys 0.027s
I20250112 08:57:04.243549 22556 raft_consensus.cc:2238] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:04.244462 22556 pending_rounds.cc:62] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:04.244696 22556 pending_rounds.cc:69] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409416030679040 op_type: NO_OP noop_request { }
I20250112 08:57:04.245018 22556 raft_consensus.cc:2883] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:04.245332 22556 raft_consensus.cc:2267] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:04.248728 22556 ts_tablet_manager.cc:1905] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:04.264308 22456 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.004s sys 0.004s
W20250112 08:57:04.264633 22456 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.597s user 0.004s sys 0.004s
I20250112 08:57:04.266575 22456 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:04.266973 22456 ts_tablet_manager.cc:1397] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.942s user 0.012s sys 0.005s
I20250112 08:57:04.280807 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 113) took 3032 ms. Trace:
I20250112 08:57:04.281244 22565 rpcz_store.cc:276] 0112 08:57:01.248605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.248876 (+ 271us) service_pool.cc:225] Handling call
0112 08:57:03.819379 (+2570503us) ts_tablet_manager.cc:1145] Deleting tablet c08341b5ce6040d5be30f324d0373e38
0112 08:57:03.819425 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:03.888656 (+ 69231us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:03.894882 (+ 6226us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.280743 (+385861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2566083}
I20250112 08:57:04.282423 22565 tablet_replica.cc:331] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:04.284808 22458 tablet_bootstrap.cc:654] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.294359 22565 raft_consensus.cc:2238] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:04.296886 22461 raft_consensus.cc:357] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.297600 22461 raft_consensus.cc:383] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.297859 22461 raft_consensus.cc:738] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.298528 22461 consensus_queue.cc:260] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.286316 22666 raft_consensus.cc:513] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.306820 22666 leader_election.cc:290] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:04.323105 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 94591.9us
W20250112 08:57:04.323875 22002 consensus_peers.cc:487] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:04.324363 22002 leader_election.cc:343] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:04.301038 22461 ts_tablet_manager.cc:1428] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.063s user 0.006s sys 0.001s
I20250112 08:57:04.331370 22461 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:04.338606 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:04.349800 22981 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:04.351827 22353 raft_consensus.cc:3054] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.352154 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:04.352257 22290 leader_election.cc:304] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
W20250112 08:57:04.355214 22149 leader_election.cc:343] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:04.355907 22002 leader_election.cc:343] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:04.360000 22832 raft_consensus.cc:2743] T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:04.197281 21713 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_0cd1e966451c4f1c9b258c3053f4a337 [id=847a5348757c4e10877a3ea2416cc60b] alter complete (version 3)
I20250112 08:57:04.372282 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bac902842453ba9d4ede5e8688f76 (DEFAULT_TABLE table=default.Table_9adfc21f03654f96a0f9e342c9177c83 [id=dd65bb8025a84452bd06a93435f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.341183 22456 raft_consensus.cc:357] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.378863 22456 raft_consensus.cc:383] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.379154 22456 raft_consensus.cc:738] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.379556 21712 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b61929ef8e8748c69fcf6bd20bf1c67d" } new_table_name: "default.Table_e61de8eab47c47008e26794edc339a29" modify_external_catalogs: true
I20250112 08:57:04.380043 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_76460b64ca214fe8b8431848c40de005"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:04.381614 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76460b64ca214fe8b8431848c40de005 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:04.381985 21716 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_71bfb1b1b306446fbe0386f94c93805f" } new_table_name: "default.Table_b4a3c3b699c348f98b156a1e95692b79" modify_external_catalogs: true
I20250112 08:57:04.383394 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_a8c02759441c4fafab2f0bf93cacee5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:04.384899 21717 master_service.cc:946] ReplaceTablet: received request to replace tablet 5c3e34e1d70a4c63956a5be660e02b21 from {username='slave'} at 127.0.0.1:56650
W20250112 08:57:04.384936 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8c02759441c4fafab2f0bf93cacee5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:04.388370 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_3aca3290d30e47068741a0ca9da50910"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:04.389864 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aca3290d30e47068741a0ca9da50910 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:04.392253 21708 master_service.cc:946] ReplaceTablet: received request to replace tablet 5157642d667d49aaad41df56be38801f from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:04.398211 22887 consensus_queue.cc:1035] T da0c101de2544cc9907c78fdf1a4af82 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:04.399084 21712 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd228f7a432b436581368da7eab3ef16 consensus state has no leader
I20250112 08:57:04.409204 22557 ts_tablet_manager.cc:1918] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:04.409571 22557 log.cc:1198] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/8c934e1a62444e64911a279f6a544c96
I20250112 08:57:04.410265 22557 ts_tablet_manager.cc:1939] T 8c934e1a62444e64911a279f6a544c96 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:04.421422 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_33baad429b774edba41b1431372b3a44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:04.423275 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33baad429b774edba41b1431372b3a44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:04.424400 22456 consensus_queue.cc:260] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.427737 22456 ts_tablet_manager.cc:1428] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.160s user 0.006s sys 0.001s
W20250112 08:57:04.437873 21716 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da16d10565a74243a9ab8c5264bf7b1a consensus state has no leader
I20250112 08:57:04.438808 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 5157642d667d49aaad41df56be38801f
I20250112 08:57:04.440383 21708 catalog_manager.cc:6596] ReplaceTablet: tablet 5157642d667d49aaad41df56be38801f deleted and replaced by tablet bbdd48de9ed74a788694e9b9ba6e3fff
I20250112 08:57:04.445169 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 122) took 3025 ms. Trace:
I20250112 08:57:04.445494 22557 rpcz_store.cc:276] 0112 08:57:01.419361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.419504 (+ 143us) service_pool.cc:225] Handling call
0112 08:57:03.960016 (+2540512us) ts_tablet_manager.cc:1145] Deleting tablet 8c934e1a62444e64911a279f6a544c96
0112 08:57:03.960058 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.177692 (+217634us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.408985 (+231293us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.443720 (+ 34735us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b5800001898. stack: 00007f9dae50b853 00007f9dae50b750 00007f9dae080584 00000000004fcd24 00000000004fccb9 00007f9dae3812d1 00007f9dae500291 00007f9dae5008c2 00007f9db7192999 00007f9db63f75f8 00007f9db64bbea2 00007f9db64ebca0 00007f9db64d2e6d 00007f9db64d2e19 00007f9db64d2da9 00007f9db64d2d71
0112 08:57:04.445110 (+ 1390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69375872,"tablet-delete.queue_time_us":2539936}
I20250112 08:57:04.452416 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bac902842453ba9d4ede5e8688f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.457294 22887 raft_consensus.cc:491] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.457870 22887 raft_consensus.cc:513] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.460412 22887 leader_election.cc:290] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:04.468484 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_794e751834834ecebf5515268c2a7eb1" } modify_external_catalogs: true
I20250112 08:57:04.469077 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_794e751834834ecebf5515268c2a7eb1" } modify_external_catalogs: true
I20250112 08:57:04.471927 22968 raft_consensus.cc:491] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.471877 21711 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_397822bc7cf943e7a8ab1fcd91cd6a31" } modify_external_catalogs: true
I20250112 08:57:04.472404 21711 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_397822bc7cf943e7a8ab1fcd91cd6a31" } modify_external_catalogs: true
I20250112 08:57:04.472389 22968 raft_consensus.cc:513] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.474645 22968 leader_election.cc:290] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:04.466423 22873 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:04.483167 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:04.331468 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 221496us
I20250112 08:57:04.486912 21714 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 1 got 0
I20250112 08:57:04.491144 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:04.496567 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:04.497329 22211 raft_consensus.cc:3054] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.498319 22212 raft_consensus.cc:2463] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:04.500897 22456 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:57:04.501734 22146 leader_election.cc:343] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:04.509503 21999 leader_election.cc:304] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:04.513015 21714 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:04.509733 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157642d667d49aaad41df56be38801f with delete_type TABLET_DATA_DELETED (replaced by tablet bbdd48de9ed74a788694e9b9ba6e3fff) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:04.516646 22968 raft_consensus.cc:2798] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:04.515774 21714 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:04.516984 22968 raft_consensus.cc:491] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:04.517309 22968 raft_consensus.cc:3054] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.517585 21714 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e19979ef29944405b833bd6dad6f3664 (table default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf]). Expected version 2 got 0
I20250112 08:57:04.518078 21714 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:04.527191 22923 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:04.511035 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:04.531780 22976 consensus_queue.cc:1035] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:04.551527 22968 raft_consensus.cc:513] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.553740 22461 tablet_bootstrap.cc:654] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.554505 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:04.555105 22213 raft_consensus.cc:3054] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.559269 22211 raft_consensus.cc:2463] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:04.569418 22213 raft_consensus.cc:2463] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:04.570426 21999 leader_election.cc:304] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:04.571177 22977 raft_consensus.cc:2798] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:04.571595 22977 raft_consensus.cc:695] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:04.572326 22977 consensus_queue.cc:237] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:04.576346 22149 leader_election.cc:343] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:04.576833 22149 leader_election.cc:304] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:04.577890 22887 raft_consensus.cc:2743] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:04.586261 22556 ts_tablet_manager.cc:1918] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:04.586791 22556 log.cc:1198] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/a090d258647b4dd7bd91cf64011735dd
I20250112 08:57:04.587185 22456 tablet_bootstrap.cc:654] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.588874 22458 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:04.589340 22458 ts_tablet_manager.cc:1397] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.554s user 0.010s sys 0.012s
I20250112 08:57:04.590633 21999 leader_election.cc:304] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:04.591621 22975 raft_consensus.cc:2798] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:04.592181 22975 raft_consensus.cc:695] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:04.593833 22556 ts_tablet_manager.cc:1939] T a090d258647b4dd7bd91cf64011735dd P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:04.593716 22975 consensus_queue.cc:237] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:04.594990 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 155728us
I20250112 08:57:04.596817 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet c08341b5ce6040d5be30f324d0373e38 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]) successfully deleted
I20250112 08:57:04.600581 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 133) took 2588 ms. Trace:
I20250112 08:57:04.600977 22556 rpcz_store.cc:276] 0112 08:57:02.011797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.020018 (+ 8221us) service_pool.cc:225] Handling call
0112 08:57:04.088168 (+2068150us) ts_tablet_manager.cc:1145] Deleting tablet a090d258647b4dd7bd91cf64011735dd
0112 08:57:04.088202 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.505826 (+417624us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.586035 (+ 80209us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.600510 (+ 14475us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154462,"tablet-delete.queue_time_us":2067529}
I20250112 08:57:04.611976 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet 253b5b6ea6f044fc8dcaafdc2e587f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:04.613073 22045 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:04.613646 22461 tablet_bootstrap.cc:492] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:04.614537 22461 ts_tablet_manager.cc:1397] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.283s user 0.009s sys 0.004s
I20250112 08:57:04.614624 22045 tablet_service.cc:1467] Processing CreateTablet for tablet 305aedf43e7241228ba88c90c329369d (DEFAULT_TABLE table=default.Table_222acc98e7144002a882838fb95ebf8e [id=82f928fe03234defb7dcd2a0e40b3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:04.615707 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305aedf43e7241228ba88c90c329369d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:04.616133 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet a090d258647b4dd7bd91cf64011735dd (table default.Table_cf768e89c13043eb9fcc7d154cd14518 [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
I20250112 08:57:04.617547 22832 raft_consensus.cc:491] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.617959 22832 raft_consensus.cc:513] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.619798 22832 leader_election.cc:290] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:04.622522 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:04.624848 22556 tablet_replica.cc:331] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:04.625578 22556 raft_consensus.cc:2238] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:04.626214 22556 pending_rounds.cc:62] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:04.626441 22556 pending_rounds.cc:69] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409425920151552 op_type: NO_OP noop_request { }
I20250112 08:57:04.626785 22556 raft_consensus.cc:2883] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:04.627103 22556 raft_consensus.cc:2267] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:04.632153 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 169529us
I20250112 08:57:04.632650 22456 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:04.633074 22456 ts_tablet_manager.cc:1397] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.000s
I20250112 08:57:04.635358 22456 raft_consensus.cc:357] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:04.655644 22456 raft_consensus.cc:383] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.660015 22456 raft_consensus.cc:738] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.660682 22456 consensus_queue.cc:260] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:04.670042 22456 ts_tablet_manager.cc:1428] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.037s user 0.004s sys 0.002s
I20250112 08:57:04.672091 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 5c3e34e1d70a4c63956a5be660e02b21
I20250112 08:57:04.673399 21717 catalog_manager.cc:6596] ReplaceTablet: tablet 5c3e34e1d70a4c63956a5be660e02b21 deleted and replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa
I20250112 08:57:04.684612 22456 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:04.684628 22565 raft_consensus.cc:2267] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:04.698683 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 58439.7us
I20250112 08:57:04.707475 22353 raft_consensus.cc:3054] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:04.707613 22557 tablet_replica.cc:331] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:04.713060 22565 ts_tablet_manager.cc:1905] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:04.713574 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:04.734158 22964 raft_consensus.cc:491] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.735826 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157642d667d49aaad41df56be38801f with delete_type TABLET_DATA_DELETED (replaced by tablet bbdd48de9ed74a788694e9b9ba6e3fff) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:04.740978 22964 raft_consensus.cc:513] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.743520 22964 leader_election.cc:290] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:04.718274 22456 tablet_bootstrap.cc:654] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.756011 22968 leader_election.cc:290] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:04.708500 22557 raft_consensus.cc:2238] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:04.756966 22557 pending_rounds.cc:62] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Trying to abort 1 pending ops.
I20250112 08:57:04.757184 22557 pending_rounds.cc:69] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113409426621468672 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "69651ab160db41b3a187542ac1cd2839" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" dest_uuid: "8c608785d6874abf9cc28db11a979ddd" new_extra_config { } }
I20250112 08:57:04.758910 22557 raft_consensus.cc:2267] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:04.591666 22458 raft_consensus.cc:357] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.760402 22458 raft_consensus.cc:383] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.760712 22458 raft_consensus.cc:738] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.761337 22458 consensus_queue.cc:260] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.771159 22565 ts_tablet_manager.cc:1918] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:04.771536 22565 log.cc:1198] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/93be4e0fc31c429f860c1a0cec64849b
I20250112 08:57:04.777695 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]) successfully deleted
I20250112 08:57:04.781018 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:04.781169 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:04.786110 22456 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:04.786574 22456 ts_tablet_manager.cc:1397] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.001s
I20250112 08:57:04.787218 21711 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet f46707de727a49108afb12ea2cf08aa0
I20250112 08:57:04.789479 22964 raft_consensus.cc:491] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.789633 22456 raft_consensus.cc:357] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.789976 22964 raft_consensus.cc:513] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.791813 22964 leader_election.cc:290] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:04.792959 22557 ts_tablet_manager.cc:1905] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:04.796348 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:04.800537 22458 ts_tablet_manager.cc:1428] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.211s user 0.007s sys 0.000s
I20250112 08:57:04.801519 22458 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:04.790391 22456 raft_consensus.cc:383] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.816062 22456 raft_consensus.cc:738] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.808406 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19979ef29944405b833bd6dad6f3664" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:04.822484 22565 ts_tablet_manager.cc:1939] T 93be4e0fc31c429f860c1a0cec64849b P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
W20250112 08:57:04.825615 21701 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd228f7a432b436581368da7eab3ef16 consensus state has no leader
I20250112 08:57:04.827183 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:04.616770 22461 raft_consensus.cc:357] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.832311 22461 raft_consensus.cc:383] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.832615 22461 raft_consensus.cc:738] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.833786 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 114) took 3579 ms. Trace:
I20250112 08:57:04.834072 22565 rpcz_store.cc:276] 0112 08:57:01.254266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.254585 (+ 319us) service_pool.cc:225] Handling call
0112 08:57:04.282278 (+3027693us) ts_tablet_manager.cc:1145] Deleting tablet 93be4e0fc31c429f860c1a0cec64849b
0112 08:57:04.282319 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.467890 (+185571us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b58000e12b0. stack: 00007f8983942853 00007f8983942750 00007f89834b7584 00000000004fcd24 00000000004fccb9 00007f898a8426d1 00007f898a8391c3 00007f898a89caba 00007f898a8c5375 00007f898ace21ac 00007f898b8f2d5a 00007f898b922ca0 00007f898b909e6d 00007f898b909e19 00007f898b909da9 00007f898b909d71
0112 08:57:04.720069 (+252179us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.770953 (+ 50884us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.833728 (+ 62775us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65846,"spinlock_wait_cycles":80663424,"tablet-delete.queue_time_us":3027079}
I20250112 08:57:04.835464 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 115) took 3578 ms. Trace:
I20250112 08:57:04.833441 22461 consensus_queue.cc:260] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.855149 22354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1514074a19694aa0a42b968a29904c30" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:04.856967 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:04.858078 22456 consensus_queue.cc:260] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.865108 22456 ts_tablet_manager.cc:1428] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.078s user 0.006s sys 0.001s
I20250112 08:57:04.865878 22456 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:04.868685 21714 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_8e7ddc505dc344c09a21b83bc0aeefce [id=b9ecc50f7e5a454394c47cf7d253ec79] alter complete (version 1)
I20250112 08:57:04.888242 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:04.888923 21713 catalog_manager.cc:5526] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:04.890231 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
W20250112 08:57:04.903656 22942 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 18160666196141519087) tablet 00000000000000000000000000000000 cur holder 0x7b5800061200
I20250112 08:57:04.910835 22557 ts_tablet_manager.cc:1918] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:04.911186 22557 log.cc:1198] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/69651ab160db41b3a187542ac1cd2839
W20250112 08:57:04.913954 22149 leader_election.cc:343] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:04.915879 22149 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:04.922456 22998 raft_consensus.cc:491] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.923338 22998 raft_consensus.cc:513] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:57:04.931805 22973 log.cc:926] Time spent T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Append to log took a long time: real 0.209s user 0.002s sys 0.000s
W20250112 08:57:04.933789 22985 log.cc:926] Time spent T da0c101de2544cc9907c78fdf1a4af82 P 8c608785d6874abf9cc28db11a979ddd: Append to log took a long time: real 0.267s user 0.000s sys 0.002s
I20250112 08:57:04.937387 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_0159265c659d4a8981db80776c88f10b" } new_table_name: "default.Table_d1591f699c154e6f9124d57a6779e410" modify_external_catalogs: true
I20250112 08:57:04.937660 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_577c2d513b7d486fbb267c9d623fb5aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:04.939102 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_577c2d513b7d486fbb267c9d623fb5aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:04.939880 21715 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_6ad283a4f2094ace8a0022618ac2bd5b" } new_table_name: "default.Table_c7e1f3d4703d41ffb340c43933e0fd41" modify_external_catalogs: true
I20250112 08:57:04.940426 22456 tablet_bootstrap.cc:654] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.961674 22557 ts_tablet_manager.cc:1939] T 69651ab160db41b3a187542ac1cd2839 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:04.964572 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 128) took 3260 ms. Trace:
I20250112 08:57:04.964985 22557 rpcz_store.cc:276] 0112 08:57:01.704636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.705124 (+ 488us) service_pool.cc:225] Handling call
0112 08:57:04.707448 (+3002324us) ts_tablet_manager.cc:1145] Deleting tablet 69651ab160db41b3a187542ac1cd2839
0112 08:57:04.707493 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.885418 (+177925us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.910621 (+ 25203us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:04.964489 (+ 53868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3001077}
I20250112 08:57:04.966378 22557 tablet_replica.cc:331] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:04.967073 22557 raft_consensus.cc:2238] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:04.970427 22456 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:04.973484 22461 ts_tablet_manager.cc:1428] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.359s user 0.006s sys 0.000s
I20250112 08:57:04.974133 22456 ts_tablet_manager.cc:1397] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.003s
I20250112 08:57:04.978467 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 54b34b09f56d4417a3f321599170e5fd
I20250112 08:57:04.982357 22185 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b34b09f56d4417a3f321599170e5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:04.973902 22556 ts_tablet_manager.cc:1905] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:04.979420 22832 raft_consensus.cc:491] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:04.989064 22832 raft_consensus.cc:513] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.990967 22832 leader_election.cc:290] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:04.980641 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b34b09f56d4417a3f321599170e5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:04.995152 22456 raft_consensus.cc:357] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:04.995936 22456 raft_consensus.cc:383] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:04.996209 22456 raft_consensus.cc:738] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:04.996927 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:04.996879 22456 consensus_queue.cc:260] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.003463 22456 ts_tablet_manager.cc:1428] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250112 08:57:05.004334 22456 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:05.008369 22557 pending_rounds.cc:62] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Trying to abort 1 pending ops.
I20250112 08:57:05.008630 22557 pending_rounds.cc:69] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409425891643392 op_type: NO_OP noop_request { }
I20250112 08:57:05.021420 22557 raft_consensus.cc:2883] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:05.021773 22557 raft_consensus.cc:2267] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:05.028981 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_200b4e56135b458abe1077676ceaba3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:05.031059 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bda308e111924297bc1d064e44647a8c" } modify_external_catalogs: true
I20250112 08:57:05.029078 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_22ea9696b04f4e90bb971e1e6cc88953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:05.031635 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bda308e111924297bc1d064e44647a8c" } modify_external_catalogs: true
W20250112 08:57:05.032399 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22ea9696b04f4e90bb971e1e6cc88953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:05.033535 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_200b4e56135b458abe1077676ceaba3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.042353 22461 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.062450 22557 ts_tablet_manager.cc:1905] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:05.069094 22290 consensus_peers.cc:487] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:05.071455 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_c1a22dbcea954ac1884b4f750e841968"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.073153 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1a22dbcea954ac1884b4f750e841968 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.095189 22461 tablet_bootstrap.cc:654] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:05.110201 21715 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 consensus state has no leader
I20250112 08:57:05.113637 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6bac902842453ba9d4ede5e8688f76 (DEFAULT_TABLE table=default.Table_9adfc21f03654f96a0f9e342c9177c83 [id=dd65bb8025a84452bd06a93435f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.115026 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b6bac902842453ba9d4ede5e8688f76. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:05.115406 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74437c3dea6547f3b9545da3f8f971bb consensus state has no leader
I20250112 08:57:05.076012 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.120429 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1514074a19694aa0a42b968a29904c30" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.121241 21713 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_00d2c67bdfbd47a6a03b2fb22f72f193 [id=78a10e1e86624d089f2b6866a4a2da03] alter complete (version 1)
W20250112 08:57:05.130259 22002 leader_election.cc:343] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.135092 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:05.144505 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.146644 22046 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.147562 22047 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b34b09f56d4417a3f321599170e5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.148334 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 93be4e0fc31c429f860c1a0cec64849b (table default.Table_8dd89a8dac4a4501a9cbe3108faab913 [id=2c86b027468242f694274397ec685267]) successfully deleted
I20250112 08:57:05.158314 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:05.159215 22048 tablet_service.cc:1467] Processing CreateTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 (DEFAULT_TABLE table=default.Table_2b22da2a0636476d9ba405517af8d41c [id=3357a5e7910d4489a32169e40b98a5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.160172 22042 tablet_service.cc:1514] Processing DeleteTablet for tablet 5157642d667d49aaad41df56be38801f with delete_type TABLET_DATA_DELETED (replaced by tablet bbdd48de9ed74a788694e9b9ba6e3fff) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.059540 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 56b9c882084142118fe251e6540fd8be (DEFAULT_TABLE table=default.Table_bc2783a8f8df477e9555dfddfef9a722 [id=cde1ceb16c4741b7abdd6f2362454467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.169699 22456 tablet_bootstrap.cc:654] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:04.835736 22565 rpcz_store.cc:276] 0112 08:57:01.257361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.257502 (+ 141us) service_pool.cc:225] Handling call
0112 08:57:04.835164 (+3577662us) ts_tablet_manager.cc:1145] Deleting tablet a44cad3bd5a84e75a41733d8d2f345a8
0112 08:57:04.835201 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:04.835421 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576696}
I20250112 08:57:05.173695 22565 tablet_replica.cc:331] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:05.174528 22044 tablet_service.cc:1467] Processing CreateTablet for tablet c49ec037c22a42108ecb48bc8270191f (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.175022 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 7aabb08ff90f42d186487dc5c99644d2 (DEFAULT_TABLE table=default.Table_a0fa6d4216744e2786892721357a3278 [id=a22d56347987446e84ea2a0dfc9a3ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.174526 22458 tablet_bootstrap.cc:654] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.188412 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_1acedaae5cd2449d9d9cbc5a91994176" } modify_external_catalogs: true
I20250112 08:57:05.189008 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_1acedaae5cd2449d9d9cbc5a91994176" } modify_external_catalogs: true
I20250112 08:57:05.176441 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 1869844a4d2b4a019dadb26d5908b520 (DEFAULT_TABLE table=default.Table_f0dab317898745629f19ef4a1712abe3 [id=482a04e7a8854fd3b42e28da88c8186f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.191504 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4fc4d2d0eaa47f6a02f518828253057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.193112 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44cad3bd5a84e75a41733d8d2f345a8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.194137 22974 consensus_queue.cc:1035] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250112 08:57:05.180976 22461 tablet_bootstrap.cc:492] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
W20250112 08:57:05.197638 22146 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.199652 22461 ts_tablet_manager.cc:1397] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.000s
I20250112 08:57:05.204069 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet c08341b5ce6040d5be30f324d0373e38 with delete_type TABLET_DATA_DELETED (replaced by tablet 815d246b7d414aadbb69ac692a5d40c7) from {username='slave'} at 127.0.0.1:56484
W20250112 08:57:05.208006 22002 leader_election.cc:343] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.210127 22461 raft_consensus.cc:357] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.210853 22461 raft_consensus.cc:383] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.211118 22461 raft_consensus.cc:738] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.213174 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49ec037c22a42108ecb48bc8270191f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.198169 22146 leader_election.cc:304] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:05.221380 22146 leader_election.cc:343] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.221796 22146 leader_election.cc:304] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.222860 22887 raft_consensus.cc:2743] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.223420 22887 raft_consensus.cc:2743] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.231187 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 80c6ea87589249739461e629eee5f40b
I20250112 08:57:05.234622 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c6ea87589249739461e629eee5f40b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:05.244480 21999 consensus_peers.cc:487] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:05.247645 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aabb08ff90f42d186487dc5c99644d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.249640 22456 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:05.250029 22456 ts_tablet_manager.cc:1397] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.246s user 0.011s sys 0.003s
I20250112 08:57:05.252313 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574999cd047643d3840a16f2baf6e9bb" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:05.254541 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet a44cad3bd5a84e75a41733d8d2f345a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a44cad3bd5a84e75a41733d8d2f345a8 already in progress: creating tablet
I20250112 08:57:05.254978 22998 leader_election.cc:290] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:05.255282 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c6ea87589249739461e629eee5f40b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.256108 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1869844a4d2b4a019dadb26d5908b520. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:05.254741 22146 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.261168 22556 ts_tablet_manager.cc:1918] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:05.262043 22456 raft_consensus.cc:357] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.263029 22456 raft_consensus.cc:383] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.263366 22456 raft_consensus.cc:738] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.264076 22456 consensus_queue.cc:260] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.266364 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88d2f7ac1d4471887c12e9e65e7a105" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.271410 22556 log.cc:1198] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:05.276772 22556 ts_tablet_manager.cc:1939] T dabc37f08d1948118b98ba25b689bea4 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:05.267796 22998 consensus_queue.cc:1035] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:05.211733 22461 consensus_queue.cc:260] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.283389 22461 ts_tablet_manager.cc:1428] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20250112 08:57:05.284188 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9c882084142118fe251e6540fd8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.287715 22456 ts_tablet_manager.cc:1428] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20250112 08:57:05.288669 22456 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:05.289417 22261 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
I20250112 08:57:05.286068 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 134) took 3274 ms. Trace:
I20250112 08:57:05.301151 22556 rpcz_store.cc:276] 0112 08:57:02.012107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.021299 (+ 9192us) service_pool.cc:225] Handling call
0112 08:57:04.624702 (+2603403us) ts_tablet_manager.cc:1145] Deleting tablet dabc37f08d1948118b98ba25b689bea4
0112 08:57:04.624744 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.097264 (+472520us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.260946 (+163682us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.286015 (+ 25069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2602749}
I20250112 08:57:05.302423 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 138) took 3029 ms. Trace:
I20250112 08:57:05.302752 22556 rpcz_store.cc:276] 0112 08:57:02.273246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.276024 (+ 2778us) service_pool.cc:225] Handling call
0112 08:57:05.302144 (+3026120us) ts_tablet_manager.cc:1145] Deleting tablet c08341b5ce6040d5be30f324d0373e38
0112 08:57:05.302205 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.302370 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3025603}
I20250112 08:57:05.304883 22461 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.313885 23009 raft_consensus.cc:491] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:05.314205 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_47c9d8c18de54c368d05e5bde3538f56" } new_table_name: "default.Table_1972c3e02af94e419ca99e14b22a365a" modify_external_catalogs: true
I20250112 08:57:05.314332 23009 raft_consensus.cc:513] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.316560 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b648fe65974c47f085a8cf94713be26e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.318176 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b648fe65974c47f085a8cf94713be26e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.318485 23009 leader_election.cc:290] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:05.321403 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_76460b64ca214fe8b8431848c40de005" } new_table_name: "default.Table_188b87023f3845a7b684e1311ec1f40e" modify_external_catalogs: true
I20250112 08:57:05.337218 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b38deeff1964b09938a12cfd4992ec5" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.343395 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 139) took 3066 ms. Trace:
I20250112 08:57:05.343780 22556 rpcz_store.cc:276] 0112 08:57:02.276771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.290174 (+ 13403us) service_pool.cc:225] Handling call
0112 08:57:05.342992 (+3052818us) ts_tablet_manager.cc:1145] Deleting tablet 92a22f87fbbf48e88dbd5af418ee94ad
0112 08:57:05.343040 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.343332 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3002498}
I20250112 08:57:05.354046 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 142) took 3007 ms. Trace:
I20250112 08:57:05.354379 22556 rpcz_store.cc:276] 0112 08:57:02.346789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.347988 (+ 1199us) service_pool.cc:225] Handling call
0112 08:57:05.353690 (+3005702us) ts_tablet_manager.cc:1145] Deleting tablet 6c34fc4b6aaf45fdae76cf153129f2da
0112 08:57:05.353735 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.353993 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3005163}
I20250112 08:57:05.355507 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 147) took 2267 ms. Trace:
I20250112 08:57:05.359450 22456 tablet_bootstrap.cc:654] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:05.365427 22146 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:05.388837 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 240898us
I20250112 08:57:05.389779 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet dabc37f08d1948118b98ba25b689bea4 (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]) successfully deleted
W20250112 08:57:05.390833 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet c08341b5ce6040d5be30f324d0373e38 because the tablet was not found. No further retry: Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38
W20250112 08:57:05.391831 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 92a22f87fbbf48e88dbd5af418ee94ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a22f87fbbf48e88dbd5af418ee94ad already in progress: creating tablet
W20250112 08:57:05.392819 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 6c34fc4b6aaf45fdae76cf153129f2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c34fc4b6aaf45fdae76cf153129f2da already in progress: creating tablet
I20250112 08:57:05.395115 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet b88d2f7ac1d4471887c12e9e65e7a105 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:05.395439 22290 leader_election.cc:304] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:05.395763 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c6ea87589249739461e629eee5f40b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:05.396790 22290 leader_election.cc:304] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:05.397439 23011 raft_consensus.cc:2743] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.398000 23011 consensus_queue.cc:1035] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:05.399358 23011 raft_consensus.cc:2743] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.400204 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44cad3bd5a84e75a41733d8d2f345a8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:05.400826 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b38deeff1964b09938a12cfd4992ec5" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:05.355824 22556 rpcz_store.cc:276] 0112 08:57:03.087704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.088003 (+ 299us) service_pool.cc:225] Handling call
0112 08:57:05.355198 (+2267195us) ts_tablet_manager.cc:1145] Deleting tablet fc82026fcd1541418176a015ea3253cf
0112 08:57:05.355237 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.355459 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266563}
I20250112 08:57:05.408761 22461 tablet_bootstrap.cc:654] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.412628 22974 raft_consensus.cc:491] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:05.413058 22974 raft_consensus.cc:513] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.414990 22974 leader_election.cc:290] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:05.415928 22403 heartbeater.cc:346] Connected to a master server at 127.21.42.252:43161
W20250112 08:57:05.417224 21999 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:05.432080 22998 raft_consensus.cc:491] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:05.432585 22998 raft_consensus.cc:513] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.434866 22998 leader_election.cc:290] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:05.437538 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.438099 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b4a3c3b699c348f98b156a1e95692b79" } modify_external_catalogs: true
I20250112 08:57:05.438232 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6d33d0d0c665414d9943c4c5f96c08c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:05.438613 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b4a3c3b699c348f98b156a1e95692b79" } modify_external_catalogs: true
W20250112 08:57:05.438865 22146 leader_election.cc:343] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:05.440510 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d33d0d0c665414d9943c4c5f96c08c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:05.445279 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648856d5b4cf45899f20c1f788dd3c66 has no consensus state
I20250112 08:57:05.449055 22565 raft_consensus.cc:2238] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:05.449211 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 148) took 2361 ms. Trace:
I20250112 08:57:05.449548 22556 rpcz_store.cc:276] 0112 08:57:03.087971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.088066 (+ 95us) service_pool.cc:225] Handling call
0112 08:57:05.408123 (+2320057us) ts_tablet_manager.cc:1145] Deleting tablet f9835288a7974e85821b095c03e91886
0112 08:57:05.445660 (+ 37537us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.449141 (+ 3481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319114}
I20250112 08:57:05.449982 22565 raft_consensus.cc:2267] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:05.450081 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f60d1f4bed4721b73deb9ba787417c" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:05.451879 22565 ts_tablet_manager.cc:1905] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:05.454829 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 340f93f630a34b3b9a9386999e2684e8 consensus state has no leader
I20250112 08:57:05.459856 22356 raft_consensus.cc:3054] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:05.464532 22149 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:05.464705 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet fc82026fcd1541418176a015ea3253cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc82026fcd1541418176a015ea3253cf already in progress: creating tablet
I20250112 08:57:05.464962 22149 leader_election.cc:304] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.465586 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:44720
W20250112 08:57:05.466030 22149 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:05.466425 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet f9835288a7974e85821b095c03e91886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9835288a7974e85821b095c03e91886 already in progress: creating tablet
I20250112 08:57:05.466374 22149 leader_election.cc:304] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.467446 22964 raft_consensus.cc:2743] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.467991 22964 raft_consensus.cc:2743] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:05.468993 22149 consensus_peers.cc:487] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870 -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:05.475167 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:05.476163 22556 tablet_replica.cc:331] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:05.477033 22556 raft_consensus.cc:2238] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:05.477807 22556 pending_rounds.cc:62] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:05.478101 22556 pending_rounds.cc:69] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409428913528832 op_type: NO_OP noop_request { }
I20250112 08:57:05.478526 22556 raft_consensus.cc:2883] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:05.479080 22556 raft_consensus.cc:2267] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:05.482270 22149 leader_election.cc:343] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.482723 22149 leader_election.cc:304] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.483572 23009 raft_consensus.cc:2743] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.486110 22461 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:05.486490 22461 ts_tablet_manager.cc:1397] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.182s user 0.009s sys 0.001s
I20250112 08:57:05.489161 22461 raft_consensus.cc:357] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:05.501869 22002 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.508513 22456 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:05.508931 22456 ts_tablet_manager.cc:1397] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.220s user 0.007s sys 0.003s
I20250112 08:57:05.511238 22456 raft_consensus.cc:357] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.502346 22002 leader_election.cc:304] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.519052 22968 raft_consensus.cc:2743] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.520783 22556 ts_tablet_manager.cc:1905] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:05.526178 22456 raft_consensus.cc:383] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.526480 22456 raft_consensus.cc:738] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.527101 22456 consensus_queue.cc:260] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.529639 22456 ts_tablet_manager.cc:1428] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250112 08:57:05.489945 22461 raft_consensus.cc:383] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.531287 22461 raft_consensus.cc:738] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.532069 22461 consensus_queue.cc:260] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.533195 22887 raft_consensus.cc:491] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:05.533596 22887 raft_consensus.cc:513] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.534760 22461 ts_tablet_manager.cc:1428] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20250112 08:57:05.535400 22887 leader_election.cc:290] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:05.535621 22461 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.560263 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:05.561659 22149 leader_election.cc:343] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.530550 22456 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:05.577755 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_ee92de1efccf428da7c2c5e6204cd4b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.579209 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee92de1efccf428da7c2c5e6204cd4b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.587313 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_8d4b7f1dc132430d8f41153f7c9c58bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.588882 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d4b7f1dc132430d8f41153f7c9c58bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.608494 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.615017 22565 ts_tablet_manager.cc:1918] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:05.615393 22565 log.cc:1198] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/8c934e1a62444e64911a279f6a544c96
W20250112 08:57:05.619155 22146 leader_election.cc:343] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:05.619748 22146 leader_election.cc:304] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:05.621294 23009 raft_consensus.cc:2743] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:05.625636 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:05.628604 22185 tablet_service.cc:1467] Processing CreateTablet for tablet f6241252631a4646812ed6ecfbf7ce6d (DEFAULT_TABLE table=default.Table_33baad429b774edba41b1431372b3a44 [id=40fc657c9ad84b6eb1b077e4ea0d9839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.629081 22324 tablet_service.cc:1467] Processing CreateTablet for tablet f6241252631a4646812ed6ecfbf7ce6d (DEFAULT_TABLE table=default.Table_33baad429b774edba41b1431372b3a44 [id=40fc657c9ad84b6eb1b077e4ea0d9839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.629609 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6241252631a4646812ed6ecfbf7ce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.630133 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6241252631a4646812ed6ecfbf7ce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.639181 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:05.642277 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 591040f7b9f9451a9979f51b2721b7a4 (DEFAULT_TABLE table=default.Table_a8c02759441c4fafab2f0bf93cacee5b [id=c76dbbefec7540bc9669fee99c56086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.641214 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:05.643328 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591040f7b9f9451a9979f51b2721b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.644379 21712 catalog_manager.cc:5526] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:05.645653 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e19979ef29944405b833bd6dad6f3664 (table default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf]). Expected version 2 got 0
I20250112 08:57:05.648471 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cf768e89c13043eb9fcc7d154cd14518" } new_table_name: "default.Table_88d7d1b5011d47be821f6e0e76f02d3e" modify_external_catalogs: true
I20250112 08:57:05.649756 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_0470958d2a10492bbc3bf701381a4095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.651194 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0470958d2a10492bbc3bf701381a4095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.657919 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_f43c29fa6315404ca33380d0034bf1b1" } new_table_name: "default.Table_b36df5002f114aceb067f4c79bc6c9ff" modify_external_catalogs: true
I20250112 08:57:05.672355 22458 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:05.673029 22458 ts_tablet_manager.cc:1397] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.872s user 0.013s sys 0.008s
I20250112 08:57:05.646414 21712 catalog_manager.cc:5526] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:05.676949 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:05.670450 22039 tablet_service.cc:1467] Processing CreateTablet for tablet f6241252631a4646812ed6ecfbf7ce6d (DEFAULT_TABLE table=default.Table_33baad429b774edba41b1431372b3a44 [id=40fc657c9ad84b6eb1b077e4ea0d9839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.679486 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6241252631a4646812ed6ecfbf7ce6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.680599 22458 raft_consensus.cc:357] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:05.681236 22458 raft_consensus.cc:383] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.681521 22458 raft_consensus.cc:738] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.693432 22557 ts_tablet_manager.cc:1918] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:05.693871 22557 log.cc:1198] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/b88d2f7ac1d4471887c12e9e65e7a105
I20250112 08:57:05.694721 22557 ts_tablet_manager.cc:1939] T b88d2f7ac1d4471887c12e9e65e7a105 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:05.696830 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_ee0e54fab7e542e2b0fde5a6dbb75bcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.698348 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee0e54fab7e542e2b0fde5a6dbb75bcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.701182 22565 ts_tablet_manager.cc:1939] T 8c934e1a62444e64911a279f6a544c96 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:05.703123 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 591040f7b9f9451a9979f51b2721b7a4 (DEFAULT_TABLE table=default.Table_a8c02759441c4fafab2f0bf93cacee5b [id=c76dbbefec7540bc9669fee99c56086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.706632 21714 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 1514074a19694aa0a42b968a29904c30
I20250112 08:57:05.709197 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:05.710711 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet da16d10565a74243a9ab8c5264bf7b1a
I20250112 08:57:05.710685 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:05.712867 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:05.712982 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:05.583755 22461 tablet_bootstrap.cc:654] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.619820 22456 tablet_bootstrap.cc:654] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:05.726259 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 267ab3e5955b4d3bbeca44baf021b328 consensus state has no leader
I20250112 08:57:05.702000 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 591040f7b9f9451a9979f51b2721b7a4 (DEFAULT_TABLE table=default.Table_a8c02759441c4fafab2f0bf93cacee5b [id=c76dbbefec7540bc9669fee99c56086a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.740794 22556 ts_tablet_manager.cc:1918] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 08:57:05.755524 22002 consensus_peers.cc:487] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:05.718150 23021 tablet.cc:1721] T 7ad9b1593a24424989f380811954439e P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:05.713712 23022 tablet.cc:1721] T 7ad9b1593a24424989f380811954439e P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:05.714586 22038 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.716980 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.720355 22458 consensus_queue.cc:260] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:05.763468 22458 ts_tablet_manager.cc:1428] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.090s user 0.004s sys 0.001s
I20250112 08:57:05.766865 22037 tablet_service.cc:1467] Processing CreateTablet for tablet da160ac36c664c26974f74070c9c2764 (DEFAULT_TABLE table=default.Table_3aca3290d30e47068741a0ca9da50910 [id=731a31c103de4103a0dbd87a343f7f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.766014 22326 tablet_service.cc:1467] Processing CreateTablet for tablet da160ac36c664c26974f74070c9c2764 (DEFAULT_TABLE table=default.Table_3aca3290d30e47068741a0ca9da50910 [id=731a31c103de4103a0dbd87a343f7f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.771468 22556 log.cc:1198] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:05.769150 22461 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:05.771962 22461 ts_tablet_manager.cc:1397] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.000s
I20250112 08:57:05.772249 22556 ts_tablet_manager.cc:1939] T 14c7056aaf8844da915acec5f9f92eeb P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:05.773139 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 116) took 4353 ms. Trace:
I20250112 08:57:05.773451 22565 rpcz_store.cc:276] 0112 08:57:01.419732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.419861 (+ 129us) service_pool.cc:225] Handling call
0112 08:57:05.173369 (+3753508us) ts_tablet_manager.cc:1145] Deleting tablet 8c934e1a62444e64911a279f6a544c96
0112 08:57:05.173448 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.516602 (+343154us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.614814 (+ 98212us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.773091 (+158277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":274473,"tablet-delete.queue_time_us":3752974}
I20250112 08:57:05.774101 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 146) took 2686 ms. Trace:
I20250112 08:57:05.774565 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 8c934e1a62444e64911a279f6a544c96 (table default.Table_2c562866672a4bbfa557feddf8f6464e [id=8196e2c3275a4c3d9ce4a6ed3b997448]) successfully deleted
I20250112 08:57:05.774411 22556 rpcz_store.cc:276] 0112 08:57:03.087435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.091989 (+ 4554us) service_pool.cc:225] Handling call
0112 08:57:05.450440 (+2358451us) ts_tablet_manager.cc:1145] Deleting tablet 14c7056aaf8844da915acec5f9f92eeb
0112 08:57:05.474142 (+ 23702us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.607855 (+133713us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.740558 (+132703us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.774047 (+ 33489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357971}
I20250112 08:57:05.775496 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 14c7056aaf8844da915acec5f9f92eeb (table default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]) successfully deleted
I20250112 08:57:05.775606 22565 tablet_replica.cc:331] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:05.776690 22565 raft_consensus.cc:2238] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:05.777186 22456 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:05.777612 22456 ts_tablet_manager.cc:1397] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.000s
I20250112 08:57:05.777526 22565 pending_rounds.cc:62] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Trying to abort 1 pending ops.
I20250112 08:57:05.780290 22456 raft_consensus.cc:357] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.781023 22456 raft_consensus.cc:383] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.781296 22456 raft_consensus.cc:738] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.781919 22456 consensus_queue.cc:260] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.785859 22556 tablet_replica.cc:331] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:05.786448 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591040f7b9f9451a9979f51b2721b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.786697 22556 raft_consensus.cc:2238] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:05.787173 22556 raft_consensus.cc:2267] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:05.789217 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cd402674080464cb3bde3697ac23180 consensus state has no leader
I20250112 08:57:05.792183 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da160ac36c664c26974f74070c9c2764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.794386 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:05.801591 22187 tablet_service.cc:1467] Processing CreateTablet for tablet da160ac36c664c26974f74070c9c2764 (DEFAULT_TABLE table=default.Table_3aca3290d30e47068741a0ca9da50910 [id=731a31c103de4103a0dbd87a343f7f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.805774 22182 tablet_service.cc:1467] Processing CreateTablet for tablet 9d669b1a721a41f9b9902c57ee8c33aa (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.803438 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.808432 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c934e1a62444e64911a279f6a544c96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:05.815630 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_dc6d3da1d6ab4e00856f33176400bc3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:05.817466 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591040f7b9f9451a9979f51b2721b7a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:05.817340 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc6d3da1d6ab4e00856f33176400bc3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.823446 22182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d669b1a721a41f9b9902c57ee8c33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.827101 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:05.808591 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 648856d5b4cf45899f20c1f788dd3c66 (DEFAULT_TABLE table=default.Table_188b87023f3845a7b684e1311ec1f40e [id=57706565a12746a9a25850bc74d7cde5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.828704 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648856d5b4cf45899f20c1f788dd3c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.832114 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_713a879ad53d408b90cdfad8f6d01a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.833890 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_713a879ad53d408b90cdfad8f6d01a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.834033 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da160ac36c664c26974f74070c9c2764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.850255 22456 ts_tablet_manager.cc:1428] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.072s user 0.008s sys 0.000s
I20250112 08:57:05.850023 22976 consensus_queue.cc:1035] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:57:05.772271 22326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da160ac36c664c26974f74070c9c2764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.856493 23005 raft_consensus.cc:491] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:05.856930 23005 raft_consensus.cc:513] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.857458 22976 consensus_queue.cc:1035] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:05.858847 23005 leader_election.cc:290] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:05.861371 22002 consensus_peers.cc:487] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:05.861716 22461 raft_consensus.cc:357] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.862360 22461 raft_consensus.cc:383] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.862622 22461 raft_consensus.cc:738] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:57:05.862944 22290 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:05.863157 22461 consensus_queue.cc:260] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.864598 22456 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:05.865164 22461 ts_tablet_manager.cc:1428] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.093s user 0.005s sys 0.000s
I20250112 08:57:05.866433 22324 tablet_service.cc:1467] Processing CreateTablet for tablet 648856d5b4cf45899f20c1f788dd3c66 (DEFAULT_TABLE table=default.Table_188b87023f3845a7b684e1311ec1f40e [id=57706565a12746a9a25850bc74d7cde5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.867455 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648856d5b4cf45899f20c1f788dd3c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.873116 22461 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.879770 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 574999cd047643d3840a16f2baf6e9bb (table default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2]). Expected version 1 got 0
I20250112 08:57:05.851171 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 132) took 4114 ms. Trace:
I20250112 08:57:05.881657 22557 rpcz_store.cc:276] 0112 08:57:01.736298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.736422 (+ 124us) service_pool.cc:225] Handling call
0112 08:57:04.966228 (+3229806us) ts_tablet_manager.cc:1145] Deleting tablet b88d2f7ac1d4471887c12e9e65e7a105
0112 08:57:04.966273 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.306040 (+339767us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.693193 (+387153us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:05.851111 (+157918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3229264}
I20250112 08:57:05.884786 22461 tablet_bootstrap.cc:654] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.884958 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 9d669b1a721a41f9b9902c57ee8c33aa (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.888953 22556 ts_tablet_manager.cc:1905] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:05.892903 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 136) took 4035 ms. Trace:
I20250112 08:57:05.893214 22557 rpcz_store.cc:276] 0112 08:57:01.857888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.858010 (+ 122us) service_pool.cc:225] Handling call
0112 08:57:05.882685 (+4024675us) ts_tablet_manager.cc:1145] Deleting tablet 35f60d1f4bed4721b73deb9ba787417c
0112 08:57:05.882731 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.892846 (+ 10115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023989}
I20250112 08:57:05.894332 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 140) took 3613 ms. Trace:
I20250112 08:57:05.894629 22557 rpcz_store.cc:276] 0112 08:57:02.280997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.285666 (+ 4669us) service_pool.cc:225] Handling call
0112 08:57:05.894020 (+3608354us) ts_tablet_manager.cc:1145] Deleting tablet 92a22f87fbbf48e88dbd5af418ee94ad
0112 08:57:05.894062 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.894285 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589683}
I20250112 08:57:05.899250 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 9d669b1a721a41f9b9902c57ee8c33aa (DEFAULT_TABLE table=default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:05.904189 22942 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 18160666196141519087) tablet 00000000000000000000000000000000 cur holder 0x7b5800061200
I20250112 08:57:05.906190 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d669b1a721a41f9b9902c57ee8c33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.914767 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ce6f55c444fe0852ad72cc7383cc1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:05.915385 22063 raft_consensus.cc:2463] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:05.916379 22290 leader_election.cc:304] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:05.917467 23005 raft_consensus.cc:2798] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:05.917785 23005 raft_consensus.cc:491] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:05.918062 23005 raft_consensus.cc:3054] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:05.923121 22461 tablet_bootstrap.cc:492] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:05.923550 22461 ts_tablet_manager.cc:1397] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.000s
I20250112 08:57:05.925560 23005 raft_consensus.cc:513] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.926585 22461 raft_consensus.cc:357] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.927196 22461 raft_consensus.cc:383] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:05.927446 22461 raft_consensus.cc:738] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:05.928104 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ce6f55c444fe0852ad72cc7383cc1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:05.927979 22461 consensus_queue.cc:260] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:05.928663 22068 raft_consensus.cc:3054] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:05.931154 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d669b1a721a41f9b9902c57ee8c33aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.895682 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 144) took 3531 ms. Trace:
I20250112 08:57:05.932055 22557 rpcz_store.cc:276] 0112 08:57:02.364396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.366156 (+ 1760us) service_pool.cc:225] Handling call
0112 08:57:05.895371 (+3529215us) ts_tablet_manager.cc:1145] Deleting tablet 6c34fc4b6aaf45fdae76cf153129f2da
0112 08:57:05.895407 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.895633 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3528661}
I20250112 08:57:05.933231 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 147) took 3097 ms. Trace:
I20250112 08:57:05.935225 22068 raft_consensus.cc:2463] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:05.936277 22290 leader_election.cc:304] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:05.936920 23011 raft_consensus.cc:2798] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:05.937043 22461 ts_tablet_manager.cc:1428] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250112 08:57:05.882278 21711 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:05.948033 22557 rpcz_store.cc:276] 0112 08:57:02.835355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.835470 (+ 115us) service_pool.cc:225] Handling call
0112 08:57:05.932925 (+3097455us) ts_tablet_manager.cc:1145] Deleting tablet fc82026fcd1541418176a015ea3253cf
0112 08:57:05.932966 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.933182 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096865}
I20250112 08:57:05.949164 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 148) took 3110 ms. Trace:
I20250112 08:57:05.949440 22557 rpcz_store.cc:276] 0112 08:57:02.838985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.839102 (+ 117us) service_pool.cc:225] Handling call
0112 08:57:05.948858 (+3109756us) ts_tablet_manager.cc:1145] Deleting tablet f9835288a7974e85821b095c03e91886
0112 08:57:05.948898 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.949114 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3109211}
I20250112 08:57:05.950285 22557 tablet_replica.cc:331] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:05.900283 22456 tablet_bootstrap.cc:654] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.950366 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
W20250112 08:57:05.904834 21704 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5cd402674080464cb3bde3697ac23180 (table default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5cd402674080464cb3bde3697ac23180 consensus state has no leader
I20250112 08:57:05.905036 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:05.952343 22324 tablet_service.cc:1467] Processing CreateTablet for tablet bbdd48de9ed74a788694e9b9ba6e3fff (DEFAULT_TABLE table=default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.953392 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd48de9ed74a788694e9b9ba6e3fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.882957 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet b88d2f7ac1d4471887c12e9e65e7a105 (table default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]) successfully deleted
W20250112 08:57:05.956143 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 73200.5us
I20250112 08:57:05.956564 23005 leader_election.cc:290] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:05.957629 22458 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:05.959762 22192 tablet_service.cc:1467] Processing CreateTablet for tablet bbdd48de9ed74a788694e9b9ba6e3fff (DEFAULT_TABLE table=default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.960772 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd48de9ed74a788694e9b9ba6e3fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:05.960649 22040 tablet_service.cc:1467] Processing CreateTablet for tablet bbdd48de9ed74a788694e9b9ba6e3fff (DEFAULT_TABLE table=default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:05.961622 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd48de9ed74a788694e9b9ba6e3fff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:05.964205 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 35f60d1f4bed4721b73deb9ba787417c because the tablet was not found. No further retry: Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c
W20250112 08:57:05.964838 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 92a22f87fbbf48e88dbd5af418ee94ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a22f87fbbf48e88dbd5af418ee94ad already in progress: creating tablet
W20250112 08:57:05.965445 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 6c34fc4b6aaf45fdae76cf153129f2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c34fc4b6aaf45fdae76cf153129f2da already in progress: creating tablet
I20250112 08:57:05.959648 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 648856d5b4cf45899f20c1f788dd3c66 (DEFAULT_TABLE table=default.Table_188b87023f3845a7b684e1311ec1f40e [id=57706565a12746a9a25850bc74d7cde5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:05.966132 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet fc82026fcd1541418176a015ea3253cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc82026fcd1541418176a015ea3253cf already in progress: creating tablet
I20250112 08:57:05.966455 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648856d5b4cf45899f20c1f788dd3c66. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:05.966663 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet f9835288a7974e85821b095c03e91886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9835288a7974e85821b095c03e91886 already in progress: creating tablet
I20250112 08:57:05.968348 22461 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.970937 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_22ea9696b04f4e90bb971e1e6cc88953" } new_table_name: "default.Table_12aae9ea91844438ab606321f803f2eb" modify_external_catalogs: true
I20250112 08:57:05.973562 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_44de87cd557a4c878db8d37210254226"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:05.975116 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44de87cd557a4c878db8d37210254226 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:05.976780 21711 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_da91c42fe80845d5b255227302fa7244" } modify_external_catalogs: true
I20250112 08:57:05.977700 22461 tablet_bootstrap.cc:654] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:05.977818 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_250ba1babd9c419e8933849397cdff93" } modify_external_catalogs: true
I20250112 08:57:05.989584 21713 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665] alter complete (version 2)
I20250112 08:57:06.000808 22461 tablet_bootstrap.cc:492] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.001199 22461 ts_tablet_manager.cc:1397] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.004s
I20250112 08:57:06.002673 23011 raft_consensus.cc:695] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:06.003614 22461 raft_consensus.cc:357] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.004340 22461 raft_consensus.cc:383] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.004606 22461 raft_consensus.cc:738] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.006821 22461 consensus_queue.cc:260] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.009070 22461 ts_tablet_manager.cc:1428] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 08:57:06.023741 22461 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:05.777885 22565 pending_rounds.cc:69] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113409426621468672 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "69651ab160db41b3a187542ac1cd2839" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d" dest_uuid: "8c608785d6874abf9cc28db11a979ddd" new_extra_config { } }
I20250112 08:57:06.026346 22565 raft_consensus.cc:2267] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.028712 22461 tablet_bootstrap.cc:654] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.032459 21711 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_da91c42fe80845d5b255227302fa7244" } modify_external_catalogs: true
I20250112 08:57:06.039712 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_250ba1babd9c419e8933849397cdff93" } modify_external_catalogs: true
W20250112 08:57:06.044137 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e57005a13bf845079bcd18784c6cab0a has no consensus state
I20250112 08:57:06.044508 22461 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.044883 22461 ts_tablet_manager.cc:1397] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20250112 08:57:06.047036 22461 raft_consensus.cc:357] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.047708 22461 raft_consensus.cc:383] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.047993 22461 raft_consensus.cc:738] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.049031 22557 raft_consensus.cc:2238] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.003541 23011 consensus_queue.cc:237] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.050671 22557 raft_consensus.cc:2267] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.051537 22461 consensus_queue.cc:260] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.061518 22461 ts_tablet_manager.cc:1428] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250112 08:57:06.062350 22461 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.067512 22461 tablet_bootstrap.cc:654] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.068554 22557 ts_tablet_manager.cc:1905] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.075661 22565 ts_tablet_manager.cc:1905] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.085500 22461 tablet_bootstrap.cc:492] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.085889 22461 ts_tablet_manager.cc:1397] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20250112 08:57:06.086337 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet ea15d9405b9944db8e00121965b9ffde
I20250112 08:57:06.088176 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet ea15d9405b9944db8e00121965b9ffde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:06.088673 22461 raft_consensus.cc:357] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.089278 22461 raft_consensus.cc:383] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.089504 22461 raft_consensus.cc:738] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.090052 22461 consensus_queue.cc:260] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.092305 22461 ts_tablet_manager.cc:1428] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250112 08:57:06.093046 22461 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.093412 21711 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:06.095139 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:06.100701 22556 ts_tablet_manager.cc:1918] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.107270 22458 tablet_bootstrap.cc:654] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.107664 22461 tablet_bootstrap.cc:654] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.114485 22556 log.cc:1198] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/253b5b6ea6f044fc8dcaafdc2e587f84
I20250112 08:57:06.120165 22968 raft_consensus.cc:491] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.120754 22968 raft_consensus.cc:513] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.124904 22968 leader_election.cc:290] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.126384 22556 ts_tablet_manager.cc:1939] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.128257 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:06.130837 22002 leader_election.cc:343] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.135236 22461 tablet_bootstrap.cc:492] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.135617 22461 ts_tablet_manager.cc:1397] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.006s
I20250112 08:57:06.136618 22039 tablet_service.cc:1514] Processing DeleteTablet for tablet ea15d9405b9944db8e00121965b9ffde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:06.137857 22461 raft_consensus.cc:357] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.138604 22461 raft_consensus.cc:383] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.138852 22461 raft_consensus.cc:738] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.139554 22461 consensus_queue.cc:260] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.140347 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_362daa956768401d9c96b43a10dbc12e" } new_table_name: "default.Table_72e464e993c04f199942af992f07f98d" modify_external_catalogs: true
I20250112 08:57:06.142123 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 158) took 2802 ms. Trace:
I20250112 08:57:06.142562 22461 ts_tablet_manager.cc:1428] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250112 08:57:06.143347 22461 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.147842 23029 raft_consensus.cc:491] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.148340 23029 raft_consensus.cc:513] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.150408 23029 leader_election.cc:290] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.152292 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:06.153620 22290 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.157227 22039 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:06.160650 22461 tablet_bootstrap.cc:654] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.142423 22556 rpcz_store.cc:276] 0112 08:57:03.339508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.340467 (+ 959us) service_pool.cc:225] Handling call
0112 08:57:05.785694 (+2445227us) ts_tablet_manager.cc:1145] Deleting tablet 253b5b6ea6f044fc8dcaafdc2e587f84
0112 08:57:05.785741 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:05.931280 (+145539us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.100489 (+169209us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.142070 (+ 41581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2444643}
I20250112 08:57:06.182749 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (table default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]) successfully deleted
I20250112 08:57:06.184136 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 159) took 2844 ms. Trace:
I20250112 08:57:06.184641 22556 rpcz_store.cc:276] 0112 08:57:03.339785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.345219 (+ 5434us) service_pool.cc:225] Handling call
0112 08:57:06.183685 (+2838466us) ts_tablet_manager.cc:1145] Deleting tablet 0b38deeff1964b09938a12cfd4992ec5
0112 08:57:06.183741 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.184070 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730038}
W20250112 08:57:06.186072 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 0b38deeff1964b09938a12cfd4992ec5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b38deeff1964b09938a12cfd4992ec5 already in progress: creating tablet
I20250112 08:57:06.186913 22556 tablet_replica.cc:331] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:06.190339 22458 tablet_bootstrap.cc:492] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:06.190724 22458 ts_tablet_manager.cc:1397] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.426s user 0.004s sys 0.004s
I20250112 08:57:06.194675 23022 tablet.cc:1721] T 574999cd047643d3840a16f2baf6e9bb P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:06.094269 22998 raft_consensus.cc:491] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.202944 22998 raft_consensus.cc:513] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.204772 22998 leader_election.cc:290] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.205174 22998 raft_consensus.cc:491] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.205577 22998 raft_consensus.cc:513] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.211544 22208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:06.213225 22208 raft_consensus.cc:2463] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:06.215255 21999 leader_election.cc:304] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:06.216531 22290 leader_election.cc:343] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:06.217579 22208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:06.218112 22208 raft_consensus.cc:2463] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:06.220454 22290 leader_election.cc:304] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:06.222056 23011 raft_consensus.cc:2798] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:06.223057 23011 raft_consensus.cc:491] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:06.226333 22968 raft_consensus.cc:2798] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:06.226649 22968 raft_consensus.cc:491] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:06.226934 22968 raft_consensus.cc:3054] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:06.228626 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b62bae16ab43414c8d8be2766802d4c7 consensus state has no leader
I20250112 08:57:06.232261 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.223459 23011 raft_consensus.cc:3054] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:06.233831 22146 leader_election.cc:343] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.235342 22998 leader_election.cc:290] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.236692 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_1871a9c14e194ccfae3a16776b48ee2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:06.238138 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1871a9c14e194ccfae3a16776b48ee2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.240823 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:06.243055 22002 consensus_peers.cc:487] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253b5b6ea6f044fc8dcaafdc2e587f84. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:06.245584 23045 tablet.cc:1721] T 574999cd047643d3840a16f2baf6e9bb P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:06.247201 21717 master_service.cc:946] ReplaceTablet: received request to replace tablet ea15d9405b9944db8e00121965b9ffde from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:06.247828 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b639de28dc9642f29a1f0caeecc05a57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:06.249521 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b639de28dc9642f29a1f0caeecc05a57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.265066 22968 raft_consensus.cc:513] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:57:06.269810 22146 leader_election.cc:343] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.254227 22556 raft_consensus.cc:2238] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.270797 22556 raft_consensus.cc:2267] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.276464 22461 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.276876 22461 ts_tablet_manager.cc:1397] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.134s user 0.020s sys 0.037s
I20250112 08:57:06.279589 22461 raft_consensus.cc:357] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.280241 22461 raft_consensus.cc:383] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.280516 22461 raft_consensus.cc:738] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.281044 22461 consensus_queue.cc:260] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.283600 22208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:06.284232 22208 raft_consensus.cc:3054] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.285547 22461 ts_tablet_manager.cc:1428] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250112 08:57:06.286288 22461 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.286360 22456 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:06.286696 22456 ts_tablet_manager.cc:1397] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.422s user 0.015s sys 0.004s
I20250112 08:57:06.289686 22456 raft_consensus.cc:357] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.290314 22456 raft_consensus.cc:383] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.290580 22456 raft_consensus.cc:738] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.291108 22456 consensus_queue.cc:260] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.260169 21713 master_service.cc:946] ReplaceTablet: received request to replace tablet f9835288a7974e85821b095c03e91886 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:06.261509 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_7111dd9cee36485ab448639684e10659" } new_table_name: "default.Table_275569887e7c49dd9608ddf9efc7fb39" modify_external_catalogs: true
I20250112 08:57:06.261335 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6872481b576147d695568e3d5bf35e77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:06.304240 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6872481b576147d695568e3d5bf35e77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.307729 23021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37418 (request call id 170) took 1247 ms. Trace:
I20250112 08:57:06.308152 23021 rpcz_store.cc:276] 0112 08:57:05.060107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.060655 (+ 548us) service_pool.cc:225] Handling call
0112 08:57:06.307671 (+1247016us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:05.354784 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:57:05.355102 (+ 318us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:57:05.355129 (+ 27us) alter_schema_op.cc:66] Acquired schema lock
0112 08:57:05.355188 (+ 59us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:57:05.355308 (+ 120us) alter_schema_op.cc:128] START. Timestamp: 1736672225348203
0112 08:57:05.355326 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 08:57:05.355845 (+ 519us) log.cc:844] Serialized 173 byte log entry
0112 08:57:05.653270 (+297425us) op_driver.cc:464] REPLICATION: finished
0112 08:57:05.718051 (+ 64781us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:57:05.757154 (+ 39103us) log.cc:844] Serialized 5 byte log entry
0112 08:57:06.306260 (+549106us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.306983 (+ 723us) log.cc:844] Serialized 14 byte log entry
0112 08:57:06.307550 (+ 567us) alter_schema_op.cc:73] Released schema lock
0112 08:57:06.307582 (+ 32us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64620,"prepare.queue_time_us":156073,"prepare.run_cpu_time_us":1614,"prepare.run_wall_time_us":71360,"replication_time_us":297852,"thread_start_us":276095,"threads_started":3,"wal-append.queue_time_us":68901}]]}
I20250112 08:57:06.316326 22556 ts_tablet_manager.cc:1905] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.362795 22968 leader_election.cc:290] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.365085 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1c634513e14e818601894834c41378" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:06.365746 22356 raft_consensus.cc:3054] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.366729 23011 raft_consensus.cc:513] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.367416 21713 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_12aae9ea91844438ab606321f803f2eb" } modify_external_catalogs: true
I20250112 08:57:06.367959 21713 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_12aae9ea91844438ab606321f803f2eb" } modify_external_catalogs: true
I20250112 08:57:06.314605 22208 raft_consensus.cc:2463] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:06.392848 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3bb2b43e854340fba2b4c4ad8a3da14a" } modify_external_catalogs: true
I20250112 08:57:06.393507 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_3bb2b43e854340fba2b4c4ad8a3da14a" } modify_external_catalogs: true
I20250112 08:57:06.394928 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_9b44fe9ca083478fa41484d20bf2a49d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:06.396440 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b44fe9ca083478fa41484d20bf2a49d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.405238 23029 raft_consensus.cc:491] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.406626 23029 raft_consensus.cc:513] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.408301 22461 tablet_bootstrap.cc:654] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.408602 23029 leader_election.cc:290] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.413496 23011 leader_election.cc:290] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.416329 22356 raft_consensus.cc:2463] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:06.419289 22556 ts_tablet_manager.cc:1918] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.419628 22556 log.cc:1198] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/93be4e0fc31c429f860c1a0cec64849b
I20250112 08:57:06.420313 22556 ts_tablet_manager.cc:1939] T 93be4e0fc31c429f860c1a0cec64849b P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.421895 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 167) took 2607 ms. Trace:
I20250112 08:57:06.422621 22556 rpcz_store.cc:276] 0112 08:57:03.814115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.827601 (+ 13486us) service_pool.cc:225] Handling call
0112 08:57:06.186781 (+2359180us) ts_tablet_manager.cc:1145] Deleting tablet 93be4e0fc31c429f860c1a0cec64849b
0112 08:57:06.186816 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.381672 (+194856us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.419102 (+ 37430us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.421847 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2358555}
I20250112 08:57:06.426257 22002 leader_election.cc:304] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:06.426307 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 93be4e0fc31c429f860c1a0cec64849b (table default.Table_8dd89a8dac4a4501a9cbe3108faab913 [id=2c86b027468242f694274397ec685267]) successfully deleted
I20250112 08:57:06.428586 22556 tablet_replica.cc:331] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:06.429383 22556 raft_consensus.cc:2238] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.429929 22556 raft_consensus.cc:2267] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.432190 22556 ts_tablet_manager.cc:1905] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.434094 22565 ts_tablet_manager.cc:1918] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 08:57:06.449858 21713 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e57005a13bf845079bcd18784c6cab0a
W20250112 08:57:06.468809 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 69272.2us
I20250112 08:57:06.471117 22461 tablet_bootstrap.cc:492] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.470902 21708 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_577c2d513b7d486fbb267c9d623fb5aa" } new_table_name: "default.Table_fe10565e765240e9b7dd43a3f4d2e6a6" modify_external_catalogs: true
I20250112 08:57:06.471702 22461 ts_tablet_manager.cc:1397] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.186s user 0.015s sys 0.003s
I20250112 08:57:06.350764 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet ea15d9405b9944db8e00121965b9ffde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:06.356786 22557 ts_tablet_manager.cc:1918] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:06.472621 22557 log.cc:1198] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:06.473327 22557 ts_tablet_manager.cc:1939] T 0b38deeff1964b09938a12cfd4992ec5 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:06.474386 22461 raft_consensus.cc:357] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.474996 22461 raft_consensus.cc:383] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.475229 22461 raft_consensus.cc:738] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.476472 22556 ts_tablet_manager.cc:1918] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250112 08:57:06.476754 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 158) took 2276 ms. Trace:
I20250112 08:57:06.476838 22556 log.cc:1198] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/5157642d667d49aaad41df56be38801f
I20250112 08:57:06.477037 22557 rpcz_store.cc:276] 0112 08:57:04.199867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.212669 (+ 12802us) service_pool.cc:225] Handling call
0112 08:57:05.950155 (+1737486us) ts_tablet_manager.cc:1145] Deleting tablet 0b38deeff1964b09938a12cfd4992ec5
0112 08:57:05.950190 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.156255 (+206065us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.356568 (+200313us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.476700 (+120132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68782,"tablet-delete.queue_time_us":1733980}
I20250112 08:57:06.477638 22556 ts_tablet_manager.cc:1939] T 5157642d667d49aaad41df56be38801f P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.478286 22557 tablet_replica.cc:331] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:06.479168 22557 raft_consensus.cc:2238] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.479765 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 168) took 2039 ms. Trace:
I20250112 08:57:06.480170 22556 rpcz_store.cc:276] 0112 08:57:04.440721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.465205 (+ 24484us) service_pool.cc:225] Handling call
0112 08:57:06.428133 (+1962928us) ts_tablet_manager.cc:1145] Deleting tablet 5157642d667d49aaad41df56be38801f
0112 08:57:06.428470 (+ 337us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.452230 (+ 23760us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.476267 (+ 24037us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.479710 (+ 3443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911357}
I20250112 08:57:06.481771 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 5157642d667d49aaad41df56be38801f (table default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]) successfully deleted
I20250112 08:57:06.482271 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:06.483000 22456 ts_tablet_manager.cc:1428] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.196s user 0.006s sys 0.001s
I20250112 08:57:06.483062 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 169) took 1702 ms. Trace:
I20250112 08:57:06.483866 22456 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:06.483479 22556 rpcz_store.cc:276] 0112 08:57:04.780701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.780814 (+ 113us) service_pool.cc:225] Handling call
0112 08:57:06.482746 (+1701932us) ts_tablet_manager.cc:1145] Deleting tablet 5c3e34e1d70a4c63956a5be660e02b21
0112 08:57:06.482787 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.483011 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701217}
W20250112 08:57:06.485040 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 5c3e34e1d70a4c63956a5be660e02b21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c3e34e1d70a4c63956a5be660e02b21 already in progress: creating tablet
I20250112 08:57:06.485858 22557 raft_consensus.cc:2267] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.487546 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 170) took 1691 ms. Trace:
I20250112 08:57:06.472033 23009 raft_consensus.cc:491] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.487879 22556 rpcz_store.cc:276] 0112 08:57:04.796073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.796194 (+ 121us) service_pool.cc:225] Handling call
0112 08:57:06.487113 (+1690919us) ts_tablet_manager.cc:1145] Deleting tablet f46707de727a49108afb12ea2cf08aa0
0112 08:57:06.487146 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.487491 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690394}
W20250112 08:57:06.488960 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet f46707de727a49108afb12ea2cf08aa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f46707de727a49108afb12ea2cf08aa0 already in progress: creating tablet
I20250112 08:57:06.489153 22356 raft_consensus.cc:3054] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.492971 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 0b38deeff1964b09938a12cfd4992ec5 (table default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]) successfully deleted
I20250112 08:57:06.495429 23009 raft_consensus.cc:513] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.498502 23009 leader_election.cc:290] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.502502 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:06.503082 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 171) took 1522 ms. Trace:
I20250112 08:57:06.503204 22213 raft_consensus.cc:3054] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.503441 22556 rpcz_store.cc:276] 0112 08:57:04.980352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.980480 (+ 128us) service_pool.cc:225] Handling call
0112 08:57:06.502707 (+1522227us) ts_tablet_manager.cc:1145] Deleting tablet 54b34b09f56d4417a3f321599170e5fd
0112 08:57:06.502757 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.503023 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510480}
W20250112 08:57:06.504609 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 54b34b09f56d4417a3f321599170e5fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54b34b09f56d4417a3f321599170e5fd already in progress: creating tablet
I20250112 08:57:06.504866 22557 ts_tablet_manager.cc:1905] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.481120 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:06.510401 22213 raft_consensus.cc:2463] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:06.513844 22290 leader_election.cc:304] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:06.514537 23011 raft_consensus.cc:2798] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:06.515043 23011 raft_consensus.cc:695] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:06.516007 22208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:06.515822 23011 consensus_queue.cc:237] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.517778 22456 tablet_bootstrap.cc:654] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.514231 22556 tablet_replica.cc:331] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:06.524593 22556 raft_consensus.cc:2238] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.525049 22556 raft_consensus.cc:2267] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.525822 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:06.526669 22556 ts_tablet_manager.cc:1905] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.547294 22565 log.cc:1198] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/69651ab160db41b3a187542ac1cd2839
W20250112 08:57:06.551966 22002 consensus_peers.cc:487] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:06.551841 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.555894 22458 raft_consensus.cc:357] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.556609 22458 raft_consensus.cc:383] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.556687 21716 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:06.556867 22458 raft_consensus.cc:738] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.557585 21716 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet e19979ef29944405b833bd6dad6f3664 (table default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf]). Expected version 2 got 0
I20250112 08:57:06.557411 22458 consensus_queue.cc:260] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:06.561256 21708 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2f340b047cd4112a56b0482a90c1db3 has no consensus state
I20250112 08:57:06.562426 22456 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:06.562804 22456 ts_tablet_manager.cc:1397] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.005s
W20250112 08:57:06.562815 22146 leader_election.cc:343] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.565078 22456 raft_consensus.cc:357] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.565722 22456 raft_consensus.cc:383] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.565950 22456 raft_consensus.cc:738] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.566136 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad9b1593a24424989f380811954439e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.566956 22456 consensus_queue.cc:260] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.571162 22556 ts_tablet_manager.cc:1918] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.571516 22556 log.cc:1198] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/65591a0bb44148c49715b4564dd13524
I20250112 08:57:06.572221 22556 ts_tablet_manager.cc:1939] T 65591a0bb44148c49715b4564dd13524 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.572921 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 172) took 1441 ms. Trace:
I20250112 08:57:06.572888 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:06.573207 22556 rpcz_store.cc:276] 0112 08:57:05.131804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.134899 (+ 3095us) service_pool.cc:225] Handling call
0112 08:57:06.514083 (+1379184us) ts_tablet_manager.cc:1145] Deleting tablet 65591a0bb44148c49715b4564dd13524
0112 08:57:06.514124 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.564200 (+ 50076us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.570973 (+ 6773us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.572870 (+ 1897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378487}
I20250112 08:57:06.573369 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_33acf77ade0f4b5d8e348dfe6e38b4fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:06.574222 22556 tablet_replica.cc:331] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
W20250112 08:57:06.574874 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33acf77ade0f4b5d8e348dfe6e38b4fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.578087 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 65591a0bb44148c49715b4564dd13524 (table default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]) successfully deleted
I20250112 08:57:06.574903 22556 raft_consensus.cc:2238] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.579121 22556 raft_consensus.cc:2267] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.580940 22290 leader_election.cc:304] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:06.582914 22556 ts_tablet_manager.cc:1905] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:06.582953 22290 leader_election.cc:343] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.591250 23029 raft_consensus.cc:491] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.591691 23029 raft_consensus.cc:513] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.592985 23021 tablet.cc:1721] T e19979ef29944405b833bd6dad6f3664 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:06.593528 23029 leader_election.cc:290] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.598631 22556 ts_tablet_manager.cc:1918] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.598740 21716 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_557a9076f797464d99d024ae141d3cc1 [id=73f85147bf6c40c68cc2a4de785358a2] alter complete (version 1)
I20250112 08:57:06.599040 22556 log.cc:1198] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/b88d2f7ac1d4471887c12e9e65e7a105
I20250112 08:57:06.599933 22556 ts_tablet_manager.cc:1939] T b88d2f7ac1d4471887c12e9e65e7a105 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.600625 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 173) took 1205 ms. Trace:
I20250112 08:57:06.600906 22556 rpcz_store.cc:276] 0112 08:57:05.394884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.394970 (+ 86us) service_pool.cc:225] Handling call
0112 08:57:06.574082 (+1179112us) ts_tablet_manager.cc:1145] Deleting tablet b88d2f7ac1d4471887c12e9e65e7a105
0112 08:57:06.574118 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.590831 (+ 16713us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.598419 (+ 7588us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.600576 (+ 2157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1178550}
I20250112 08:57:06.601272 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f60d1f4bed4721b73deb9ba787417c" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.602159 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 174) took 1207 ms. Trace:
I20250112 08:57:06.601943 22068 raft_consensus.cc:2463] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:06.603164 23052 raft_consensus.cc:2798] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:06.603550 22290 leader_election.cc:304] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:06.604218 23029 raft_consensus.cc:2798] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:06.604540 23029 raft_consensus.cc:491] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:06.604841 23029 raft_consensus.cc:3054] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.602831 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet b88d2f7ac1d4471887c12e9e65e7a105 (table default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]) successfully deleted
I20250112 08:57:06.606736 22458 ts_tablet_manager.cc:1428] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.416s user 0.007s sys 0.000s
I20250112 08:57:06.604267 23052 raft_consensus.cc:695] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:06.608920 23052 consensus_queue.cc:237] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.612074 23029 raft_consensus.cc:513] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:06.613829 22290 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:06.602460 22556 rpcz_store.cc:276] 0112 08:57:05.395134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.395622 (+ 488us) service_pool.cc:225] Handling call
0112 08:57:06.601850 (+1206228us) ts_tablet_manager.cc:1145] Deleting tablet 80c6ea87589249739461e629eee5f40b
0112 08:57:06.601889 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.602112 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205278}
I20250112 08:57:06.615509 22556 tablet_replica.cc:331] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:06.616223 22556 raft_consensus.cc:2238] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:06.616832 22556 pending_rounds.cc:62] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:06.617041 22556 pending_rounds.cc:69] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409438094282752 op_type: NO_OP noop_request { }
I20250112 08:57:06.617370 22556 raft_consensus.cc:2883] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:06.617676 22556 raft_consensus.cc:2267] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:06.618489 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 80c6ea87589249739461e629eee5f40b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c6ea87589249739461e629eee5f40b already in progress: creating tablet
I20250112 08:57:06.619601 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f60d1f4bed4721b73deb9ba787417c" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:06.620178 22068 raft_consensus.cc:3054] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.622063 22458 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:06.626879 22068 raft_consensus.cc:2463] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:06.628223 23029 leader_election.cc:290] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.630506 22556 ts_tablet_manager.cc:1905] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:06.631429 22290 leader_election.cc:343] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:06.632151 22290 leader_election.cc:304] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:06.638885 22458 tablet_bootstrap.cc:654] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.638850 23029 raft_consensus.cc:2798] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:06.629680 23022 tablet.cc:1721] T e19979ef29944405b833bd6dad6f3664 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:06.642616 22456 ts_tablet_manager.cc:1428] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.080s user 0.007s sys 0.000s
I20250112 08:57:06.648228 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:06.649022 22355 raft_consensus.cc:2463] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:06.650633 22456 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:06.652355 22565 ts_tablet_manager.cc:1939] T 69651ab160db41b3a187542ac1cd2839 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:06.655098 23009 raft_consensus.cc:491] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.655596 23009 raft_consensus.cc:513] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.656163 22557 ts_tablet_manager.cc:1918] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.656682 22557 log.cc:1198] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/253b5b6ea6f044fc8dcaafdc2e587f84
I20250112 08:57:06.657573 22557 ts_tablet_manager.cc:1939] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:06.658998 23009 leader_election.cc:290] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.659725 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea15d9405b9944db8e00121965b9ffde" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:06.661981 22146 leader_election.cc:343] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.669515 23021 tablet.cc:1721] T 574999cd047643d3840a16f2baf6e9bb P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:06.676692 23029 raft_consensus.cc:695] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:06.677484 23029 consensus_queue.cc:237] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:06.681322 22149 leader_election.cc:343] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.681687 22149 leader_election.cc:304] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:06.682228 23009 raft_consensus.cc:2743] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:06.682823 22556 ts_tablet_manager.cc:1918] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.683141 22556 log.cc:1198] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:06.683840 22556 ts_tablet_manager.cc:1939] T 010ce6f55c444fe0852ad72cc7383cc1 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.667183 22461 consensus_queue.cc:260] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.685941 22461 ts_tablet_manager.cc:1428] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.213s user 0.003s sys 0.001s
I20250112 08:57:06.686718 22461 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.687077 23009 raft_consensus.cc:491] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.687554 23009 raft_consensus.cc:513] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.684532 22556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 175) took 1219 ms. Trace:
I20250112 08:57:06.688525 22556 rpcz_store.cc:276] 0112 08:57:05.465303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.465418 (+ 115us) service_pool.cc:225] Handling call
0112 08:57:06.615362 (+1149944us) ts_tablet_manager.cc:1145] Deleting tablet 010ce6f55c444fe0852ad72cc7383cc1
0112 08:57:06.615401 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.637909 (+ 22508us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.682633 (+ 44724us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.684485 (+ 1852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149345}
I20250112 08:57:06.689694 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 010ce6f55c444fe0852ad72cc7383cc1 (table default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]) successfully deleted
I20250112 08:57:06.659818 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 157) took 2460 ms. Trace:
I20250112 08:57:06.691287 22456 tablet_bootstrap.cc:654] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.696197 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1889a1ab4f4d8aa4f7915d91f2cf27" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:06.697397 22146 leader_election.cc:343] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.698467 23009 leader_election.cc:290] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
W20250112 08:57:06.700435 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 1514074a19694aa0a42b968a29904c30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1514074a19694aa0a42b968a29904c30 already in progress: creating tablet
W20250112 08:57:06.701071 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet da16d10565a74243a9ab8c5264bf7b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da16d10565a74243a9ab8c5264bf7b1a already in progress: creating tablet
I20250112 08:57:06.691713 22557 rpcz_store.cc:276] 0112 08:57:04.199133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.212970 (+ 13837us) service_pool.cc:225] Handling call
0112 08:57:06.478122 (+2265152us) ts_tablet_manager.cc:1145] Deleting tablet 253b5b6ea6f044fc8dcaafdc2e587f84
0112 08:57:06.478164 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.557960 (+ 79796us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.655885 (+ 97925us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.659749 (+ 3864us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6188,"tablet-delete.queue_time_us":2241598}
W20250112 08:57:06.705277 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet ea15d9405b9944db8e00121965b9ffde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea15d9405b9944db8e00121965b9ffde already in progress: creating tablet
W20250112 08:57:06.705972 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 already in progress: creating tablet
I20250112 08:57:06.707648 22461 tablet_bootstrap.cc:654] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.718801 22461 tablet_bootstrap.cc:492] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.719255 22461 ts_tablet_manager.cc:1397] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.006s
I20250112 08:57:06.722103 22557 tablet_replica.cc:331] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:06.727737 22557 raft_consensus.cc:2238] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:06.728456 22557 raft_consensus.cc:2267] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.728108 22461 raft_consensus.cc:357] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.728766 22461 raft_consensus.cc:383] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.729000 22461 raft_consensus.cc:738] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.653120 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 122) took 4743 ms. Trace:
I20250112 08:57:06.729357 22565 rpcz_store.cc:276] 0112 08:57:01.909283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.914034 (+ 4751us) service_pool.cc:225] Handling call
0112 08:57:05.775345 (+3861311us) ts_tablet_manager.cc:1145] Deleting tablet 69651ab160db41b3a187542ac1cd2839
0112 08:57:05.775449 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.385577 (+610128us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.433889 (+ 48312us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.653072 (+219183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3853770}
I20250112 08:57:06.729674 22461 consensus_queue.cc:260] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.731859 22355 raft_consensus.cc:3054] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.735538 22461 ts_tablet_manager.cc:1428] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20250112 08:57:06.736356 22461 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.741771 22557 ts_tablet_manager.cc:1905] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.747803 22461 tablet_bootstrap.cc:654] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.757221 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (table default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]) successfully deleted
I20250112 08:57:06.759101 22461 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.758846 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_135f424a13604fd5a9500b3a4762f139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:06.759496 22461 ts_tablet_manager.cc:1397] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20250112 08:57:06.760721 23029 raft_consensus.cc:491] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.760715 21716 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_53a046eb23ec4f73b2e96111509abefe" } modify_external_catalogs: true
I20250112 08:57:06.761171 22456 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:06.761211 21716 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_53a046eb23ec4f73b2e96111509abefe" } modify_external_catalogs: true
W20250112 08:57:06.760977 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 79677.1us
I20250112 08:57:06.761145 23029 raft_consensus.cc:513] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.761605 22456 ts_tablet_manager.cc:1397] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.111s user 0.008s sys 0.005s
W20250112 08:57:06.762179 22149 leader_election.cc:343] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.763017 23029 leader_election.cc:290] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.762763 22149 leader_election.cc:304] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:06.765368 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.766044 22068 raft_consensus.cc:2463] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:06.767038 22290 leader_election.cc:304] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:06.767853 23029 raft_consensus.cc:2798] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:06.768185 23029 raft_consensus.cc:491] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:06.768517 23029 raft_consensus.cc:3054] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.772081 23045 tablet.cc:1721] T e19979ef29944405b833bd6dad6f3664 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:06.779868 22461 raft_consensus.cc:357] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.780557 22461 raft_consensus.cc:383] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.780809 22461 raft_consensus.cc:738] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.784574 21716 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:06.763878 22456 raft_consensus.cc:357] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.785522 22456 raft_consensus.cc:383] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.785805 22456 raft_consensus.cc:738] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.786180 22461 consensus_queue.cc:260] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:06.786437 22456 consensus_queue.cc:260] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.787333 23029 raft_consensus.cc:513] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.788535 22456 ts_tablet_manager.cc:1428] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20250112 08:57:06.789918 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:06.790611 22068 raft_consensus.cc:3054] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.764010 23009 raft_consensus.cc:2743] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:06.792881 22565 tablet_replica.cc:331] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:06.794548 22565 raft_consensus.cc:2238] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:06.796460 22565 raft_consensus.cc:2267] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.798456 22461 ts_tablet_manager.cc:1428] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250112 08:57:06.804912 23029 leader_election.cc:290] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.805822 22461 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.811288 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:06.812000 22210 raft_consensus.cc:2463] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:06.811767 22456 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:06.820039 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e1484cd84479abba824cc8a6fae1f" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:06.820582 22212 raft_consensus.cc:3054] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.826062 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_2055fd4bf7224371ba88109fd2e0f629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:06.829213 23009 raft_consensus.cc:491] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.829420 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_79dac8e9ea5d483397bd1ec1cbc74403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:06.830029 23009 raft_consensus.cc:513] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.833333 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:06.834815 22146 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.836455 23009 leader_election.cc:290] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.796516 22068 raft_consensus.cc:2463] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:06.838426 22290 leader_election.cc:304] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
W20250112 08:57:06.838706 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 76611.6us
I20250112 08:57:06.839250 23063 raft_consensus.cc:2798] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:06.839527 22149 leader_election.cc:304] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:06.840801 23037 raft_consensus.cc:2798] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:06.841140 23037 raft_consensus.cc:491] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:06.841431 23037 raft_consensus.cc:3054] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:06.843048 22456 tablet_bootstrap.cc:654] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.845039 23029 raft_consensus.cc:491] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:06.846530 23029 raft_consensus.cc:513] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.849125 22557 ts_tablet_manager.cc:1918] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250112 08:57:06.849509 22557 log.cc:1198] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/5157642d667d49aaad41df56be38801f
I20250112 08:57:06.850261 22557 ts_tablet_manager.cc:1939] T 5157642d667d49aaad41df56be38801f P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:06.852208 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea15d9405b9944db8e00121965b9ffde" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:06.853188 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79dac8e9ea5d483397bd1ec1cbc74403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:06.856101 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2055fd4bf7224371ba88109fd2e0f629 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.857198 22330 tablet_service.cc:1514] Processing DeleteTablet for tablet ef66f5e785864d1c845f5c5a5f944473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:06.857854 22556 tablet_replica.cc:331] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
W20250112 08:57:06.857872 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_135f424a13604fd5a9500b3a4762f139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.858529 22556 raft_consensus.cc:2238] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:06.859273 22556 raft_consensus.cc:2267] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.861050 22556 ts_tablet_manager.cc:1905] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.864384 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 166) took 2128 ms. Trace:
I20250112 08:57:06.864692 22557 rpcz_store.cc:276] 0112 08:57:04.735538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.735676 (+ 138us) service_pool.cc:225] Handling call
0112 08:57:06.711086 (+1975410us) ts_tablet_manager.cc:1145] Deleting tablet 5157642d667d49aaad41df56be38801f
0112 08:57:06.721989 (+ 10903us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.783798 (+ 61809us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.848906 (+ 65108us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.864321 (+ 15415us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4834,"tablet-delete.queue_time_us":1970234}
I20250112 08:57:06.865952 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 167) took 2085 ms. Trace:
I20250112 08:57:06.866235 22557 rpcz_store.cc:276] 0112 08:57:04.780861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.781011 (+ 150us) service_pool.cc:225] Handling call
0112 08:57:06.865630 (+2084619us) ts_tablet_manager.cc:1145] Deleting tablet 5c3e34e1d70a4c63956a5be660e02b21
0112 08:57:06.865666 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.865905 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084073}
I20250112 08:57:06.867449 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 168) took 2040 ms. Trace:
I20250112 08:57:06.867722 22557 rpcz_store.cc:276] 0112 08:57:04.826864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.827007 (+ 143us) service_pool.cc:225] Handling call
0112 08:57:06.867117 (+2040110us) ts_tablet_manager.cc:1145] Deleting tablet f46707de727a49108afb12ea2cf08aa0
0112 08:57:06.867157 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.867403 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2039414}
I20250112 08:57:06.888909 22212 raft_consensus.cc:2463] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:06.889292 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 169) took 1907 ms. Trace:
I20250112 08:57:06.889592 22557 rpcz_store.cc:276] 0112 08:57:04.982055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.982196 (+ 141us) service_pool.cc:225] Handling call
0112 08:57:06.888906 (+1906710us) ts_tablet_manager.cc:1145] Deleting tablet 54b34b09f56d4417a3f321599170e5fd
0112 08:57:06.888946 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.889223 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1906076}
I20250112 08:57:06.884415 23029 leader_election.cc:290] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:06.890491 22557 tablet_replica.cc:331] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:06.893679 22557 raft_consensus.cc:2238] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.894014 22461 tablet_bootstrap.cc:654] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.894127 22557 raft_consensus.cc:2267] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.896799 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:06.908424 22557 ts_tablet_manager.cc:1905] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.912992 22456 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:06.913429 22456 ts_tablet_manager.cc:1397] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.007s
I20250112 08:57:06.915982 22456 raft_consensus.cc:357] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.916594 22456 raft_consensus.cc:383] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.916888 22456 raft_consensus.cc:738] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.917619 22456 consensus_queue.cc:260] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.920498 22556 ts_tablet_manager.cc:1918] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:06.923157 22556 log.cc:1198] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:06.924149 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9848b879a2e40aeb840924261e6a100" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:06.924649 22556 ts_tablet_manager.cc:1939] T ef66f5e785864d1c845f5c5a5f944473 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:06.925835 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet ef66f5e785864d1c845f5c5a5f944473 (table default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]) successfully deleted
W20250112 08:57:06.928357 21703 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b62bae16ab43414c8d8be2766802d4c7 consensus state has no leader
I20250112 08:57:06.932559 22461 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.934000 22461 ts_tablet_manager.cc:1397] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.128s user 0.017s sys 0.000s
I20250112 08:57:06.936232 22290 leader_election.cc:304] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:06.937007 23029 raft_consensus.cc:2743] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:06.930054 22456 ts_tablet_manager.cc:1428] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20250112 08:57:06.939705 22461 raft_consensus.cc:357] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.940012 22456 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:06.941097 22461 raft_consensus.cc:383] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.941835 22461 raft_consensus.cc:738] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.942997 22461 consensus_queue.cc:260] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.946502 22461 ts_tablet_manager.cc:1428] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250112 08:57:06.947057 22456 tablet_bootstrap.cc:654] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.948834 22461 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.849028 23037 raft_consensus.cc:513] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.951689 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
W20250112 08:57:06.952961 22146 leader_election.cc:343] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:06.956912 21717 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_8d4b7f1dc132430d8f41153f7c9c58bc" } new_table_name: "default.Table_0a2d43ff203345df884d6761e6e56085" modify_external_catalogs: true
I20250112 08:57:06.957684 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet ef66f5e785864d1c845f5c5a5f944473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:06.960636 21716 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_b2e28d9ccf214a9aafc78b9f7e74328b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:06.961707 22461 tablet_bootstrap.cc:654] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:06.963944 21999 consensus_peers.cc:487] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253b5b6ea6f044fc8dcaafdc2e587f84. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:06.964759 21715 master_service.cc:946] ReplaceTablet: received request to replace tablet 267ab3e5955b4d3bbeca44baf021b328 from {username='slave'} at 127.0.0.1:56650
W20250112 08:57:06.965989 21716 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2e28d9ccf214a9aafc78b9f7e74328b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:06.967023 22557 ts_tablet_manager.cc:1918] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:06.967496 22557 log.cc:1198] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/80c6ea87589249739461e629eee5f40b
I20250112 08:57:06.973793 22461 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:06.974154 22461 ts_tablet_manager.cc:1397] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20250112 08:57:06.968457 23037 leader_election.cc:290] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:06.980286 22565 ts_tablet_manager.cc:1905] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:06.981701 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet ef66f5e785864d1c845f5c5a5f944473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:06.983304 23063 raft_consensus.cc:695] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:06.984400 23063 consensus_queue.cc:237] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.988401 22456 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:06.988783 22456 ts_tablet_manager.cc:1397] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.007s
I20250112 08:57:06.989756 22461 raft_consensus.cc:357] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.990466 22461 raft_consensus.cc:383] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.990962 22461 raft_consensus.cc:738] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:06.990988 22456 raft_consensus.cc:357] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.991614 22456 raft_consensus.cc:383] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:06.991724 22461 consensus_queue.cc:260] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:06.992327 22557 ts_tablet_manager.cc:1939] T 80c6ea87589249739461e629eee5f40b P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:06.993942 22461 ts_tablet_manager.cc:1428] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250112 08:57:06.994131 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 171) took 1759 ms. Trace:
I20250112 08:57:06.994256 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15d38686b2c64b20ba87275e3ba176ab. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:57:06.994417 22557 rpcz_store.cc:276] 0112 08:57:05.234388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.234503 (+ 115us) service_pool.cc:225] Handling call
0112 08:57:06.890358 (+1655855us) ts_tablet_manager.cc:1145] Deleting tablet 80c6ea87589249739461e629eee5f40b
0112 08:57:06.890395 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:06.959149 (+ 68754us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.966763 (+ 7614us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:06.994076 (+ 27313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655390}
I20250112 08:57:06.995298 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:06.995671 22557 tablet_replica.cc:331] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:06.996064 22979 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:06.996379 22461 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:06.996404 22557 raft_consensus.cc:2238] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:06.996765 22557 raft_consensus.cc:2267] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:06.996627 22979 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:06.998322 22557 ts_tablet_manager.cc:1905] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.006083 22355 raft_consensus.cc:3054] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.010706 22979 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:07.011022 22461 tablet_bootstrap.cc:654] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:06.991858 22456 raft_consensus.cc:738] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.011767 22456 consensus_queue.cc:260] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.014099 22456 ts_tablet_manager.cc:1428] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250112 08:57:07.015166 22456 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.032088 23078 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 08:57:07.036818 22461 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:07.037191 22461 ts_tablet_manager.cc:1397] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.013s
I20250112 08:57:07.041272 22456 tablet_bootstrap.cc:654] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.042943 22461 raft_consensus.cc:357] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.043601 22461 raft_consensus.cc:383] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.043838 22461 raft_consensus.cc:738] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.046475 22461 consensus_queue.cc:260] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.052254 22461 ts_tablet_manager.cc:1428] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250112 08:57:07.053040 22461 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.061221 22461 tablet_bootstrap.cc:654] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.071766 22456 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.072188 22456 ts_tablet_manager.cc:1397] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.000s
I20250112 08:57:07.073661 21802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:07.074679 22456 raft_consensus.cc:357] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.075336 22456 raft_consensus.cc:383] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 08:57:07.083020 21717 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc9e74cf00da42938661dfddb3ff8f8a has no consensus state
I20250112 08:57:07.075657 22456 raft_consensus.cc:738] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.100319 22456 consensus_queue.cc:260] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.104573 22557 ts_tablet_manager.cc:1918] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:07.105028 22557 log.cc:1198] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:07.108896 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 5157642d667d49aaad41df56be38801f (table default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]) successfully deleted
W20250112 08:57:07.109618 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 5c3e34e1d70a4c63956a5be660e02b21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c3e34e1d70a4c63956a5be660e02b21 already in progress: creating tablet
I20250112 08:57:07.109756 22557 ts_tablet_manager.cc:1939] T a44cad3bd5a84e75a41733d8d2f345a8 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:57:07.110327 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet f46707de727a49108afb12ea2cf08aa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f46707de727a49108afb12ea2cf08aa0 already in progress: creating tablet
W20250112 08:57:07.110949 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 54b34b09f56d4417a3f321599170e5fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54b34b09f56d4417a3f321599170e5fd already in progress: creating tablet
I20250112 08:57:07.111515 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 80c6ea87589249739461e629eee5f40b (table default.Table_bda308e111924297bc1d064e44647a8c [id=78bc9e684bde43269ebdb804dd657e65]) successfully deleted
I20250112 08:57:07.116880 22456 ts_tablet_manager.cc:1428] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20250112 08:57:07.118010 22456 tablet_bootstrap.cc:492] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.125166 23009 raft_consensus.cc:491] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.125592 23009 raft_consensus.cc:513] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:07.126143 22149 leader_election.cc:343] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.126513 22149 leader_election.cc:304] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.127324 23047 raft_consensus.cc:2743] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.127369 23009 leader_election.cc:290] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:07.128141 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 173) took 1489 ms. Trace:
I20250112 08:57:07.128459 22557 rpcz_store.cc:276] 0112 08:57:05.638894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.639029 (+ 135us) service_pool.cc:225] Handling call
0112 08:57:06.995520 (+1356491us) ts_tablet_manager.cc:1145] Deleting tablet a44cad3bd5a84e75a41733d8d2f345a8
0112 08:57:06.995569 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.047947 (+ 52378us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.104289 (+ 56342us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.128086 (+ 23797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355878}
I20250112 08:57:07.129472 22557 tablet_replica.cc:331] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.130265 22557 raft_consensus.cc:2238] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.130720 22557 raft_consensus.cc:2267] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.126339 22458 tablet_bootstrap.cc:492] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:07.140233 22458 ts_tablet_manager.cc:1397] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.533s user 0.005s sys 0.006s
I20250112 08:57:07.142351 22458 raft_consensus.cc:357] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.142978 22458 raft_consensus.cc:383] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.143262 22458 raft_consensus.cc:738] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.129561 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd402674080464cb3bde3697ac23180" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:07.144971 22146 leader_election.cc:343] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.146307 23063 raft_consensus.cc:491] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.150846 22351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd402674080464cb3bde3697ac23180" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.152184 22458 consensus_queue.cc:260] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.148964 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1889a1ab4f4d8aa4f7915d91f2cf27" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.156060 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet a44cad3bd5a84e75a41733d8d2f345a8 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]) successfully deleted
I20250112 08:57:07.156883 22557 ts_tablet_manager.cc:1905] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:07.163780 22149 leader_election.cc:343] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.164448 22149 leader_election.cc:304] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.152477 22354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.168829 22354 raft_consensus.cc:2463] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:07.171180 22458 ts_tablet_manager.cc:1428] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.031s user 0.001s sys 0.003s
I20250112 08:57:07.152607 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:07.172717 22357 raft_consensus.cc:3054] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.176779 22458 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:07.177533 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:07.149503 23063 raft_consensus.cc:513] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.180701 23063 leader_election.cc:290] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.185099 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:07.186698 22461 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
W20250112 08:57:07.186875 22290 leader_election.cc:343] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.187057 22461 ts_tablet_manager.cc:1397] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.134s user 0.024s sys 0.050s
I20250112 08:57:07.189636 22357 raft_consensus.cc:2463] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:07.191476 22461 raft_consensus.cc:357] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.192075 22461 raft_consensus.cc:383] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.192287 22461 raft_consensus.cc:738] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.192901 22461 consensus_queue.cc:260] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.193962 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:57:07.194938 22461 ts_tablet_manager.cc:1428] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 08:57:07.195686 22461 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.196086 22979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:07.202983 23037 raft_consensus.cc:2743] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:07.203063 22149 leader_election.cc:343] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.203478 22149 leader_election.cc:304] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.183853 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:07.204321 22210 raft_consensus.cc:2463] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:07.204895 21717 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bcb5d680dc6047ae9d719a1428694045" } modify_external_catalogs: true
I20250112 08:57:07.205405 21717 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_bcb5d680dc6047ae9d719a1428694045" } modify_external_catalogs: true
I20250112 08:57:07.205402 23045 tablet.cc:1721] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:07.205852 23037 raft_consensus.cc:2743] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.205981 23011 raft_consensus.cc:2743] T 7ad9b1593a24424989f380811954439e P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.211371 22290 leader_election.cc:304] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.211995 23070 raft_consensus.cc:491] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.212399 23070 raft_consensus.cc:513] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:07.212754 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_55a599c370f441969aad1da2ebfb4793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:07.214154 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55a599c370f441969aad1da2ebfb4793 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.214249 23070 leader_election.cc:290] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.214645 23070 raft_consensus.cc:2798] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.214910 23070 raft_consensus.cc:491] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.215193 23070 raft_consensus.cc:3054] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.216434 22456 tablet_bootstrap.cc:654] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.220350 22149 leader_election.cc:304] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.222744 23037 raft_consensus.cc:2798] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.223073 23037 raft_consensus.cc:491] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.223392 23037 raft_consensus.cc:3054] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.224349 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44cad3bd5a84e75a41733d8d2f345a8" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:07.226434 22290 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:07.227118 22290 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:07.227421 22290 leader_election.cc:304] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.228065 23063 raft_consensus.cc:2743] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.227799 22862 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "76718d1596134c83900541c6d398d59e" instance_seqno: 1736672206912728) as {username='slave'} at 127.21.42.193:50913; Asking this server to re-register.
I20250112 08:57:07.229480 22863 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "53068c99960647728141706e9c50b870" instance_seqno: 1736672209127407) as {username='slave'} at 127.21.42.194:40053; Asking this server to re-register.
I20250112 08:57:07.231395 23037 raft_consensus.cc:513] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.234241 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
W20250112 08:57:07.236341 22146 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.237684 22461 tablet_bootstrap.cc:654] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.239501 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cd402674080464cb3bde3697ac23180 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:07.240273 23070 raft_consensus.cc:513] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:07.241803 23041 log.cc:926] Time spent T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Append to log took a long time: real 0.987s user 0.023s sys 0.000s
I20250112 08:57:07.242815 23037 leader_election.cc:290] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:07.242838 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:07.241509 22149 leader_election.cc:304] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.244485 23037 raft_consensus.cc:2798] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
W20250112 08:57:07.245090 22290 leader_election.cc:343] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.245929 23037 raft_consensus.cc:695] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:07.247480 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "267ab3e5955b4d3bbeca44baf021b328" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:07.248163 22356 raft_consensus.cc:3054] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.250211 22862 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" instance_seqno: 1736672211048426) as {username='slave'} at 127.21.42.195:53517; Asking this server to re-register.
I20250112 08:57:07.254796 22356 raft_consensus.cc:2463] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:07.249292 22261 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:07.264009 22149 leader_election.cc:304] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.265512 22456 tablet_bootstrap.cc:492] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.265650 23047 raft_consensus.cc:2798] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:07.265889 22456 ts_tablet_manager.cc:1397] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.148s user 0.012s sys 0.000s
I20250112 08:57:07.266037 23047 raft_consensus.cc:695] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:07.266805 23047 consensus_queue.cc:237] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.268097 22456 raft_consensus.cc:357] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.268725 22456 raft_consensus.cc:383] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.269412 22456 raft_consensus.cc:738] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.269958 22456 consensus_queue.cc:260] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.264633 23037 consensus_queue.cc:237] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.273329 22456 ts_tablet_manager.cc:1428] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250112 08:57:07.274240 22456 tablet_bootstrap.cc:492] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.283612 22557 ts_tablet_manager.cc:1918] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:07.288560 23070 leader_election.cc:290] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.289981 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02538a000a0f4fa5a40f1ab5cd862fe1" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:07.290551 22210 raft_consensus.cc:3054] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.295836 22403 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:07.301054 22461 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:07.301470 22461 ts_tablet_manager.cc:1397] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.001s
I20250112 08:57:07.302937 22210 raft_consensus.cc:2463] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:07.303742 22461 raft_consensus.cc:357] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.304492 22461 raft_consensus.cc:383] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.304741 22461 raft_consensus.cc:738] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.305999 22461 consensus_queue.cc:260] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.308398 22290 leader_election.cc:304] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.309350 23063 raft_consensus.cc:2798] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:07.309737 23063 raft_consensus.cc:695] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:07.310003 22461 ts_tablet_manager.cc:1428] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250112 08:57:07.310803 22461 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.310444 23063 consensus_queue.cc:237] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.311125 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_6280d02d39bc47eba2925dcc1fef5835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:07.311638 22557 log.cc:1198] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/1514074a19694aa0a42b968a29904c30
W20250112 08:57:07.312592 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6280d02d39bc47eba2925dcc1fef5835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.316495 22113 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:07.316535 22557 ts_tablet_manager.cc:1939] T 1514074a19694aa0a42b968a29904c30 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:07.318663 23052 raft_consensus.cc:491] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.319100 23052 raft_consensus.cc:513] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.320971 23052 leader_election.cc:290] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:07.325011 21999 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:07.328048 23070 raft_consensus.cc:491] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.328554 23070 raft_consensus.cc:513] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.332660 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06717cb567a54ee6a268c3be5d8786ec" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.334088 22356 raft_consensus.cc:2463] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:07.335211 22002 leader_election.cc:304] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:07.335764 23052 raft_consensus.cc:491] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.336050 23033 raft_consensus.cc:2798] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.336360 23033 raft_consensus.cc:491] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.336205 23052 raft_consensus.cc:513] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.336668 23033 raft_consensus.cc:3054] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.339133 23052 leader_election.cc:290] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.340198 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:07.343719 22458 tablet_bootstrap.cc:654] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.345582 23033 raft_consensus.cc:513] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.347048 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5959caa4d39420981e3b1994df4cb17" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.347817 22356 raft_consensus.cc:2388] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 53068c99960647728141706e9c50b870 in this term.
I20250112 08:57:07.349007 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06717cb567a54ee6a268c3be5d8786ec" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:07.349627 22357 raft_consensus.cc:3054] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.352623 22002 leader_election.cc:304] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.353613 23052 raft_consensus.cc:3054] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.365777 22357 raft_consensus.cc:2463] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:07.366927 22002 leader_election.cc:304] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:07.367686 22968 raft_consensus.cc:2798] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:07.371306 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:07.371430 23070 leader_election.cc:290] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.371734 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:07.372640 22210 raft_consensus.cc:2463] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:07.373546 22458 tablet_bootstrap.cc:492] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:07.373920 22458 ts_tablet_manager.cc:1397] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.197s user 0.005s sys 0.004s
I20250112 08:57:07.373987 22290 leader_election.cc:304] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:07.374714 23070 raft_consensus.cc:2798] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.375077 23070 raft_consensus.cc:491] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.375352 23070 raft_consensus.cc:3054] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.375665 22863 ts_manager.cc:194] Registered new tserver with Master: 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:07.376711 22565 ts_tablet_manager.cc:1918] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:07.377053 22565 log.cc:1198] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/b88d2f7ac1d4471887c12e9e65e7a105
W20250112 08:57:07.377249 22290 leader_election.cc:343] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.377733 22565 ts_tablet_manager.cc:1939] T b88d2f7ac1d4471887c12e9e65e7a105 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:07.380044 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 175) took 1669 ms. Trace:
I20250112 08:57:07.380398 22557 rpcz_store.cc:276] 0112 08:57:05.710379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.710533 (+ 154us) service_pool.cc:225] Handling call
0112 08:57:07.129329 (+1418796us) ts_tablet_manager.cc:1145] Deleting tablet 1514074a19694aa0a42b968a29904c30
0112 08:57:07.129370 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.247340 (+117970us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.283405 (+ 36065us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.379980 (+ 96575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418188}
I20250112 08:57:07.381738 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 1514074a19694aa0a42b968a29904c30 (table default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]) successfully deleted
I20250112 08:57:07.384475 23033 leader_election.cc:290] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.385882 22861 ts_manager.cc:194] Registered new tserver with Master: 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:07.386873 22456 tablet_bootstrap.cc:654] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:07.385114 21999 leader_election.cc:343] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:07.385437 23052 raft_consensus.cc:2743] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.390864 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 176) took 1678 ms. Trace:
I20250112 08:57:07.391227 22557 rpcz_store.cc:276] 0112 08:57:05.712725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.712842 (+ 117us) service_pool.cc:225] Handling call
0112 08:57:07.390408 (+1677566us) ts_tablet_manager.cc:1145] Deleting tablet da16d10565a74243a9ab8c5264bf7b1a
0112 08:57:07.390475 (+ 67us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.390792 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1677052}
I20250112 08:57:07.391826 23070 raft_consensus.cc:513] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.394569 22461 tablet_bootstrap.cc:654] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.395017 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:07.395589 22210 raft_consensus.cc:3054] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:07.397831 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet da16d10565a74243a9ab8c5264bf7b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da16d10565a74243a9ab8c5264bf7b1a already in progress: creating tablet
I20250112 08:57:07.399009 22557 tablet_replica.cc:331] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.399715 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:07.404536 22557 raft_consensus.cc:2238] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.404944 22557 raft_consensus.cc:2267] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.406462 22557 ts_tablet_manager.cc:1905] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:07.406440 22290 leader_election.cc:343] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.407790 22210 raft_consensus.cc:2463] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:07.408890 22290 leader_election.cc:304] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.411106 23052 raft_consensus.cc:491] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.411697 22862 ts_manager.cc:194] Registered new tserver with Master: 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.412151 23011 raft_consensus.cc:2798] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:07.412592 23011 raft_consensus.cc:695] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:07.413393 23011 consensus_queue.cc:237] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.413847 23052 raft_consensus.cc:513] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.416433 23029 raft_consensus.cc:491] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.416924 23029 raft_consensus.cc:513] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.418932 23029 leader_election.cc:290] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.419843 22461 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:07.420217 22461 ts_tablet_manager.cc:1397] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.000s
I20250112 08:57:07.422914 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:07.421914 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:07.424391 22210 raft_consensus.cc:2463] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
W20250112 08:57:07.425120 22290 leader_election.cc:343] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.425689 22290 leader_election.cc:304] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.426539 23029 raft_consensus.cc:2798] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.426947 23029 raft_consensus.cc:491] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.427312 23029 raft_consensus.cc:3054] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.435747 23029 raft_consensus.cc:513] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.438006 22456 tablet_bootstrap.cc:492] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.438409 22456 ts_tablet_manager.cc:1397] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.002s
I20250112 08:57:07.438784 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
W20250112 08:57:07.440109 22290 leader_election.cc:343] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.440781 22456 raft_consensus.cc:357] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.441500 22456 raft_consensus.cc:383] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.441730 22456 raft_consensus.cc:738] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.442267 22456 consensus_queue.cc:260] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.445895 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:07.446486 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:07.446990 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet e19979ef29944405b833bd6dad6f3664 (table default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf]). Expected version 2 got 0
I20250112 08:57:07.446898 22458 raft_consensus.cc:357] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.447347 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7098f8f60445eb899434504ef59e4b (DEFAULT_TABLE table=default.Table_dc6d3da1d6ab4e00856f33176400bc3c [id=c18af33d73d6468cb6de121bc16a5578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.447582 22458 raft_consensus.cc:383] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.447448 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61aedea18d944179e788c08488c0a5e" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:07.447829 22458 raft_consensus.cc:738] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.447983 22212 raft_consensus.cc:3054] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.448446 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7098f8f60445eb899434504ef59e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.448451 22458 consensus_queue.cc:260] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.449820 22187 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7098f8f60445eb899434504ef59e4b (DEFAULT_TABLE table=default.Table_dc6d3da1d6ab4e00856f33176400bc3c [id=c18af33d73d6468cb6de121bc16a5578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.450819 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7098f8f60445eb899434504ef59e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.450848 22458 ts_tablet_manager.cc:1428] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.077s user 0.005s sys 0.001s
I20250112 08:57:07.451617 22458 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:07.448686 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7098f8f60445eb899434504ef59e4b (DEFAULT_TABLE table=default.Table_dc6d3da1d6ab4e00856f33176400bc3c [id=c18af33d73d6468cb6de121bc16a5578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.454332 22212 raft_consensus.cc:2463] T f61aedea18d944179e788c08488c0a5e P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:07.455331 22290 leader_election.cc:304] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.456087 23063 raft_consensus.cc:2798] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:07.456725 23063 raft_consensus.cc:695] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:07.457471 23063 consensus_queue.cc:237] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.379812 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 125) took 5469 ms. Trace:
I20250112 08:57:07.404232 23070 leader_election.cc:290] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.460260 22565 rpcz_store.cc:276] 0112 08:57:01.910142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:01.915444 (+ 5302us) service_pool.cc:225] Handling call
0112 08:57:06.792519 (+4877075us) ts_tablet_manager.cc:1145] Deleting tablet b88d2f7ac1d4471887c12e9e65e7a105
0112 08:57:06.792652 (+ 133us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.173883 (+381231us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.376491 (+202608us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.379738 (+ 3247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4834042}
I20250112 08:57:07.461372 22565 tablet_replica.cc:331] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:07.461293 23029 leader_election.cc:290] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.461678 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet b88d2f7ac1d4471887c12e9e65e7a105 (table default.Table_9420682ee5cb40e893640510c8fafecb [id=d0235e7c5bdb41cb9522a9ed750f30dd]) successfully deleted
I20250112 08:57:07.462076 22565 raft_consensus.cc:2238] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:07.462913 22565 raft_consensus.cc:2267] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:07.465262 21714 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 43) took 1070 ms (client timeout 1000 ms). Trace:
W20250112 08:57:07.465555 21714 rpcz_store.cc:269] 0112 08:57:06.394699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.394795 (+ 96us) service_pool.cc:225] Handling call
0112 08:57:07.465184 (+1070389us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
W20250112 08:57:07.466266 22290 consensus_peers.cc:487] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:07.466501 21714 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_135f424a13604fd5a9500b3a4762f139" } new_table_name: "default.Table_4198997d702f4c4bb04a3082456851c8" modify_external_catalogs: true
I20250112 08:57:07.467300 23037 raft_consensus.cc:491] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.467706 23037 raft_consensus.cc:513] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.468325 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:07.469009 21711 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:07.469586 23037 leader_election.cc:290] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:07.471144 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a22f87fbbf48e88dbd5af418ee94ad" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:07.472411 22146 leader_election.cc:343] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.474061 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a22f87fbbf48e88dbd5af418ee94ad" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.474618 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 1df6282224b849a488071323e2e69e1c
W20250112 08:57:07.475107 22149 leader_election.cc:343] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.475454 22149 leader_election.cc:304] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.475813 21712 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56650 (request call id 257) took 1083 ms. Trace:
I20250112 08:57:07.476322 23037 raft_consensus.cc:2743] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.476521 22182 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df6282224b849a488071323e2e69e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:07.477618 22038 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df6282224b849a488071323e2e69e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:07.476105 21712 rpcz_store.cc:276] 0112 08:57:06.392332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.392463 (+ 131us) service_pool.cc:225] Handling call
0112 08:57:06.393970 (+ 1507us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:57:06.507796 (+113826us) catalog_manager.cc:2753] Modifying in-memory table state
0112 08:57:06.507944 (+ 148us) catalog_manager.cc:2762] Locking tablets
0112 08:57:07.413030 (+905086us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 08:57:07.474254 (+ 61224us) catalog_manager.cc:2794] Removing table from by-name map
0112 08:57:07.474451 (+ 197us) catalog_manager.cc:2810] Committing in-memory state
0112 08:57:07.474545 (+ 94us) catalog_manager.cc:2814] Aborting table tasks
0112 08:57:07.475760 (+ 1215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:07.414912 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:07.415268 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:07.415292 (+ 24us) write_op.cc:435] Acquired schema lock
0112 08:57:07.415306 (+ 14us) tablet.cc:659] Decoding operations
0112 08:57:07.415616 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:07.415656 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:07.415679 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:57:07.415962 (+ 283us) tablet.cc:698] Row locks acquired
0112 08:57:07.415976 (+ 14us) write_op.cc:260] PREPARE: finished
0112 08:57:07.416092 (+ 116us) write_op.cc:270] Start()
0112 08:57:07.416190 (+ 98us) write_op.cc:276] Timestamp: P: 1736672227416027 usec, L: 0
0112 08:57:07.416209 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 08:57:07.416745 (+ 536us) log.cc:844] Serialized 731 byte log entry
0112 08:57:07.471642 (+ 54897us) op_driver.cc:464] REPLICATION: finished
0112 08:57:07.472418 (+ 776us) write_op.cc:301] APPLY: starting
0112 08:57:07.472479 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:07.472584 (+ 105us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:07.472679 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:07.472922 (+ 243us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:07.473058 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 08:57:07.473104 (+ 46us) write_op.cc:312] APPLY: finished
0112 08:57:07.473305 (+ 201us) log.cc:844] Serialized 29 byte log entry
0112 08:57:07.473976 (+ 671us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:07.474040 (+ 64us) write_op.cc:454] Released schema lock
0112 08:57:07.474070 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":600,"num_ops":2,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":2056,"prepare.run_wall_time_us":3340,"replication_time_us":55335}]]}
I20250112 08:57:07.479355 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df6282224b849a488071323e2e69e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:07.486675 22324 tablet_service.cc:1467] Processing CreateTablet for tablet cbeaee2ab95140f8825a6ca59c27701b (DEFAULT_TABLE table=default.Table_ee0e54fab7e542e2b0fde5a6dbb75bcc [id=c0328dc52f044212bd2266aa418de8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.487713 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbeaee2ab95140f8825a6ca59c27701b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.491972 22038 tablet_service.cc:1467] Processing CreateTablet for tablet cbeaee2ab95140f8825a6ca59c27701b (DEFAULT_TABLE table=default.Table_ee0e54fab7e542e2b0fde5a6dbb75bcc [id=c0328dc52f044212bd2266aa418de8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.493428 22182 tablet_service.cc:1467] Processing CreateTablet for tablet cbeaee2ab95140f8825a6ca59c27701b (DEFAULT_TABLE table=default.Table_ee0e54fab7e542e2b0fde5a6dbb75bcc [id=c0328dc52f044212bd2266aa418de8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.493832 21715 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 267ab3e5955b4d3bbeca44baf021b328
I20250112 08:57:07.494429 22182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbeaee2ab95140f8825a6ca59c27701b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.495072 21715 catalog_manager.cc:6596] ReplaceTablet: tablet 267ab3e5955b4d3bbeca44baf021b328 deleted and replaced by tablet 340fcbf7e3cf4038b5bed1b188da18dd
I20250112 08:57:07.497069 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 with delete_type TABLET_DATA_DELETED (replaced by tablet 340fcbf7e3cf4038b5bed1b188da18dd) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:07.482924 22456 ts_tablet_manager.cc:1428] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20250112 08:57:07.498709 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 with delete_type TABLET_DATA_DELETED (replaced by tablet 340fcbf7e3cf4038b5bed1b188da18dd) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:07.500656 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 with delete_type TABLET_DATA_DELETED (replaced by tablet 340fcbf7e3cf4038b5bed1b188da18dd) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:07.494084 22461 raft_consensus.cc:357] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.503288 22461 raft_consensus.cc:383] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.507206 23037 raft_consensus.cc:491] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.507647 23037 raft_consensus.cc:513] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.509079 22565 ts_tablet_manager.cc:1905] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.509506 23037 leader_election.cc:290] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:07.511265 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b7098f8f60445eb899434504ef59e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.511960 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e62435dce1441a3b2fee50518662d48" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:07.513125 22146 leader_election.cc:343] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.514611 23029 raft_consensus.cc:491] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.514456 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e62435dce1441a3b2fee50518662d48" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.514819 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 (DEFAULT_TABLE table=default.Table_ee92de1efccf428da7c2c5e6204cd4b5 [id=c12a2b10b35d48328c0f8f4eb656cfb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.515130 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 (DEFAULT_TABLE table=default.Table_ee92de1efccf428da7c2c5e6204cd4b5 [id=c12a2b10b35d48328c0f8f4eb656cfb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.515695 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad1f85596894fe9992ec4ad115e6928. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:07.515836 22149 leader_election.cc:343] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.516100 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad1f85596894fe9992ec4ad115e6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.516199 22149 leader_election.cc:304] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.517020 23037 raft_consensus.cc:2743] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.517678 23029 raft_consensus.cc:513] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.520146 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbeaee2ab95140f8825a6ca59c27701b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.520146 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1514074a19694aa0a42b968a29904c30" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:07.521327 22290 leader_election.cc:343] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:07.523872 22290 leader_election.cc:343] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 1514074a19694aa0a42b968a29904c30
I20250112 08:57:07.524293 22290 leader_election.cc:304] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:07.524927 23029 leader_election.cc:290] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.525669 23029 raft_consensus.cc:2743] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:07.529920 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 1df6282224b849a488071323e2e69e1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df6282224b849a488071323e2e69e1c already in progress: creating tablet
I20250112 08:57:07.530671 23103 tablet_replica.cc:331] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:07.509985 22461 raft_consensus.cc:738] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.536549 22461 consensus_queue.cc:260] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.541579 22456 tablet_bootstrap.cc:492] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.544009 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 (DEFAULT_TABLE table=default.Table_ee92de1efccf428da7c2c5e6204cd4b5 [id=c12a2b10b35d48328c0f8f4eb656cfb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.544996 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad1f85596894fe9992ec4ad115e6928. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.560915 22461 ts_tablet_manager.cc:1428] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.140s user 0.005s sys 0.001s
I20250112 08:57:07.561764 22461 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.574818 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 (DEFAULT_TABLE table=default.Table_6d33d0d0c665414d9943c4c5f96c08c2 [id=b2125ac29047476fb78de2b8db82db8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.576053 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.579840 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 (DEFAULT_TABLE table=default.Table_6d33d0d0c665414d9943c4c5f96c08c2 [id=b2125ac29047476fb78de2b8db82db8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.580852 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.582110 22456 tablet_bootstrap.cc:654] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.586587 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 (DEFAULT_TABLE table=default.Table_6d33d0d0c665414d9943c4c5f96c08c2 [id=b2125ac29047476fb78de2b8db82db8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.587647 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.603176 22557 ts_tablet_manager.cc:1918] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:07.603595 22557 log.cc:1198] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/ea15d9405b9944db8e00121965b9ffde
I20250112 08:57:07.604205 22557 ts_tablet_manager.cc:1939] T ea15d9405b9944db8e00121965b9ffde P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:07.605962 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 181) took 1256 ms. Trace:
I20250112 08:57:07.606245 22557 rpcz_store.cc:276] 0112 08:57:06.350043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.350641 (+ 598us) service_pool.cc:225] Handling call
0112 08:57:07.398867 (+1048226us) ts_tablet_manager.cc:1145] Deleting tablet ea15d9405b9944db8e00121965b9ffde
0112 08:57:07.398907 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.487872 (+ 88965us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.602941 (+115069us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.605904 (+ 2963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926232}
I20250112 08:57:07.607442 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet ea15d9405b9944db8e00121965b9ffde (table default.Table_250ba1babd9c419e8933849397cdff93 [id=a42ff5d5b1f949ef97f131233f826798]) successfully deleted
I20250112 08:57:07.609829 23033 consensus_queue.cc:1035] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:07.612519 22968 raft_consensus.cc:695] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:07.614384 22968 consensus_queue.cc:237] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.616904 23052 leader_election.cc:290] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.623965 22456 tablet_bootstrap.cc:492] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.624451 22456 ts_tablet_manager.cc:1397] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.003s
W20250112 08:57:07.626447 22002 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:07.626900 22456 raft_consensus.cc:357] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.627619 22456 raft_consensus.cc:383] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.630618 22461 tablet_bootstrap.cc:654] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.531710 23103 raft_consensus.cc:2238] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.642740 22557 tablet_replica.cc:331] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.643504 22557 raft_consensus.cc:2238] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.643937 22557 raft_consensus.cc:2267] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.645529 22557 ts_tablet_manager.cc:1905] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.646495 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_660015f62b6c4db59a358f76526c5dad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:07.647985 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_660015f62b6c4db59a358f76526c5dad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.650550 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_00c8e2048c444eeca4149e2b643a7abe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:07.651949 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00c8e2048c444eeca4149e2b643a7abe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.653792 23037 raft_consensus.cc:491] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.654189 23037 raft_consensus.cc:513] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.655951 23037 leader_election.cc:290] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:07.658270 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc82026fcd1541418176a015ea3253cf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:07.659524 22149 leader_election.cc:343] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.661468 22461 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:07.662580 22461 ts_tablet_manager.cc:1397] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.010s
I20250112 08:57:07.627843 22456 raft_consensus.cc:738] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.668570 23103 raft_consensus.cc:2267] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.668296 22456 consensus_queue.cc:260] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.672961 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 08c419a664bf4e93b8bcb6e0411ceb3c (DEFAULT_TABLE table=default.Table_200b4e56135b458abe1077676ceaba3f [id=7c08bd4ed5784c1db6462e73e9b6776a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.674036 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c419a664bf4e93b8bcb6e0411ceb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.676647 21709 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 5cd402674080464cb3bde3697ac23180
I20250112 08:57:07.677946 21709 catalog_manager.cc:6596] ReplaceTablet: tablet 5cd402674080464cb3bde3697ac23180 deleted and replaced by tablet ae66efbd303d4a25b4c547915109bca7
I20250112 08:57:07.681370 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd402674080464cb3bde3697ac23180 with delete_type TABLET_DATA_DELETED (replaced by tablet ae66efbd303d4a25b4c547915109bca7) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:07.683578 22458 tablet_bootstrap.cc:654] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:07.685051 21714 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a162200945646729e1744cd6fd407c3 has no consensus state
I20250112 08:57:07.686762 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:07.687466 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc82026fcd1541418176a015ea3253cf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:07.689473 22876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41480 (request call id 0) took 4086 ms. Trace:
I20250112 08:57:07.693606 22456 ts_tablet_manager.cc:1428] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
W20250112 08:57:07.696786 22146 leader_election.cc:343] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.697374 22146 leader_election.cc:304] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.700058 23037 raft_consensus.cc:2743] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.701365 21714 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_cb9e1fb387114cfe86c64bcdc7a1c4c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:07.701951 21713 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:07.702313 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 08c419a664bf4e93b8bcb6e0411ceb3c (DEFAULT_TABLE table=default.Table_200b4e56135b458abe1077676ceaba3f [id=7c08bd4ed5784c1db6462e73e9b6776a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:07.702785 21714 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb9e1fb387114cfe86c64bcdc7a1c4c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.703511 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c419a664bf4e93b8bcb6e0411ceb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.703598 21713 catalog_manager.cc:5526] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:07.704816 22185 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e74cf00da42938661dfddb3ff8f8a (DEFAULT_TABLE table=default.Table_0a2d43ff203345df884d6761e6e56085 [id=e52ec8fa018f459a8fcf902f517192dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.706113 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e74cf00da42938661dfddb3ff8f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.706507 21709 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_fcb376f8af89430c84a3f2c0fea366b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:07.707801 21709 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcb376f8af89430c84a3f2c0fea366b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:07.667605 22461 raft_consensus.cc:357] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.709277 22461 raft_consensus.cc:383] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.709556 22461 raft_consensus.cc:738] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.710163 22326 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e74cf00da42938661dfddb3ff8f8a (DEFAULT_TABLE table=default.Table_0a2d43ff203345df884d6761e6e56085 [id=e52ec8fa018f459a8fcf902f517192dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.710480 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd402674080464cb3bde3697ac23180 with delete_type TABLET_DATA_DELETED (replaced by tablet ae66efbd303d4a25b4c547915109bca7) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:07.712483 22456 tablet_bootstrap.cc:492] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.713106 23063 consensus_queue.cc:1035] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:07.718241 22461 consensus_queue.cc:260] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.720469 22461 ts_tablet_manager.cc:1428] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.055s user 0.005s sys 0.000s
I20250112 08:57:07.726051 22326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e74cf00da42938661dfddb3ff8f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.726565 22456 tablet_bootstrap.cc:654] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.748010 23103 ts_tablet_manager.cc:1905] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.768358 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd402674080464cb3bde3697ac23180 with delete_type TABLET_DATA_DELETED (replaced by tablet ae66efbd303d4a25b4c547915109bca7) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:07.770851 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 08c419a664bf4e93b8bcb6e0411ceb3c (DEFAULT_TABLE table=default.Table_200b4e56135b458abe1077676ceaba3f [id=7c08bd4ed5784c1db6462e73e9b6776a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.771832 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08c419a664bf4e93b8bcb6e0411ceb3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.771929 22456 tablet_bootstrap.cc:492] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
W20250112 08:57:07.771688 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 170877us
I20250112 08:57:07.772970 22456 ts_tablet_manager.cc:1397] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.005s
I20250112 08:57:07.775249 23009 raft_consensus.cc:491] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.775628 23009 raft_consensus.cc:513] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.777343 22324 tablet_service.cc:1467] Processing CreateTablet for tablet c2f340b047cd4112a56b0482a90c1db3 (DEFAULT_TABLE table=default.Table_fe10565e765240e9b7dd43a3f4d2e6a6 [id=8022b1aa8763462f90039c83a1122228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.777917 23009 leader_election.cc:290] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:07.777792 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_6d33d0d0c665414d9943c4c5f96c08c2" } modify_external_catalogs: true
I20250112 08:57:07.778306 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_6d33d0d0c665414d9943c4c5f96c08c2" } modify_external_catalogs: true
I20250112 08:57:07.779729 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:07.780164 22557 ts_tablet_manager.cc:1918] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 08:57:07.781031 22146 leader_election.cc:343] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.781599 22456 raft_consensus.cc:357] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.782310 22456 raft_consensus.cc:383] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.782604 22456 raft_consensus.cc:738] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.783212 22456 consensus_queue.cc:260] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.780485 22557 log.cc:1198] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:07.787860 22557 ts_tablet_manager.cc:1939] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:07.789846 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 182) took 1439 ms. Trace:
I20250112 08:57:07.790123 22557 rpcz_store.cc:276] 0112 08:57:06.350293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.472314 (+122021us) service_pool.cc:225] Handling call
0112 08:57:07.642588 (+1170274us) ts_tablet_manager.cc:1145] Deleting tablet ac1889a1ab4f4d8aa4f7915d91f2cf27
0112 08:57:07.642630 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.736960 (+ 94330us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.779966 (+ 43006us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:07.789797 (+ 9831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126977}
I20250112 08:57:07.794565 22456 ts_tablet_manager.cc:1428] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250112 08:57:07.795389 22456 tablet_bootstrap.cc:492] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.795074 22039 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e74cf00da42938661dfddb3ff8f8a (DEFAULT_TABLE table=default.Table_0a2d43ff203345df884d6761e6e56085 [id=e52ec8fa018f459a8fcf902f517192dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.796277 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e74cf00da42938661dfddb3ff8f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.802100 22182 tablet_service.cc:1467] Processing CreateTablet for tablet c2f340b047cd4112a56b0482a90c1db3 (DEFAULT_TABLE table=default.Table_fe10565e765240e9b7dd43a3f4d2e6a6 [id=8022b1aa8763462f90039c83a1122228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.808933 22325 tablet_service.cc:1467] Processing CreateTablet for tablet 7d730646b5b04e5eb9dd9cce1db90638 (DEFAULT_TABLE table=default.Table_c1a22dbcea954ac1884b4f750e841968 [id=adcc38f793b34989b3369c9d5e143295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.812729 22456 tablet_bootstrap.cc:654] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.815292 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (table default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]) successfully deleted
I20250112 08:57:07.803140 22182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f340b047cd4112a56b0482a90c1db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.815057 22557 tablet_replica.cc:331] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.818475 22557 raft_consensus.cc:2238] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.818907 22557 raft_consensus.cc:2267] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.824585 22557 ts_tablet_manager.cc:1905] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.834671 22456 tablet_bootstrap.cc:492] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.835062 22456 ts_tablet_manager.cc:1397] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20250112 08:57:07.845947 23009 raft_consensus.cc:491] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.846432 23009 raft_consensus.cc:513] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:07.838526 23041 log.cc:926] Time spent T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Append to log took a long time: real 0.059s user 0.000s sys 0.002s
I20250112 08:57:07.846522 22192 tablet_service.cc:1467] Processing CreateTablet for tablet 7d730646b5b04e5eb9dd9cce1db90638 (DEFAULT_TABLE table=default.Table_c1a22dbcea954ac1884b4f750e841968 [id=adcc38f793b34989b3369c9d5e143295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.848057 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d730646b5b04e5eb9dd9cce1db90638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.855073 21708 catalog_manager.cc:5526] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:07.856391 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:07.860291 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9
I20250112 08:57:07.861984 22327 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:07.879281 22327 tablet_service.cc:1467] Processing CreateTablet for tablet ccc909a01a6340dcb6902eeccf821f5d (DEFAULT_TABLE table=default.Table_b648fe65974c47f085a8cf94713be26e [id=288f1c6fb89244709fec83bbe7507602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:07.850486 23009 leader_election.cc:290] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:07.837123 22456 raft_consensus.cc:357] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.880214 22456 raft_consensus.cc:383] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.880481 22456 raft_consensus.cc:738] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.881016 22456 consensus_queue.cc:260] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.882947 22456 ts_tablet_manager.cc:1428] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20250112 08:57:07.862919 22191 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:07.890532 22456 tablet_bootstrap.cc:492] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.896279 22557 ts_tablet_manager.cc:1918] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:07.896617 22557 log.cc:1198] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:07.897303 22456 tablet_bootstrap.cc:654] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.899600 22557 ts_tablet_manager.cc:1939] T ef66f5e785864d1c845f5c5a5f944473 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:07.901942 22557 tablet_replica.cc:331] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.901742 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet ef66f5e785864d1c845f5c5a5f944473 (table default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]) successfully deleted
I20250112 08:57:07.902805 22557 raft_consensus.cc:2238] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:07.903242 22557 raft_consensus.cc:2267] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.903581 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.905067 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:07.905340 22149 leader_election.cc:343] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.905778 22456 tablet_bootstrap.cc:492] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.905730 22149 leader_election.cc:304] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:07.906134 22456 ts_tablet_manager.cc:1397] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20250112 08:57:07.906749 23009 raft_consensus.cc:2743] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:07.906854 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f340b047cd4112a56b0482a90c1db3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:07.907843 22149 leader_election.cc:343] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:07.908843 22557 ts_tablet_manager.cc:1905] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.909281 22461 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.912577 23063 consensus_queue.cc:1035] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:07.913267 22456 raft_consensus.cc:357] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.913902 22456 raft_consensus.cc:383] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.914134 22456 raft_consensus.cc:738] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.915447 22456 consensus_queue.cc:260] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.921779 23063 raft_consensus.cc:491] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.922173 23063 raft_consensus.cc:513] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.923069 22456 ts_tablet_manager.cc:1428] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250112 08:57:07.924575 23063 leader_election.cc:290] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.923882 22456 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.926251 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:07.926836 22211 raft_consensus.cc:2463] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:07.928089 22290 leader_election.cc:304] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:07.928956 23063 raft_consensus.cc:2798] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:07.929307 23063 raft_consensus.cc:491] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:07.929651 23063 raft_consensus.cc:3054] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:07.930573 22461 tablet_bootstrap.cc:654] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.931247 22456 tablet_bootstrap.cc:654] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:07.939980 23009 raft_consensus.cc:491] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:07.940459 23009 raft_consensus.cc:513] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:07.942163 23009 leader_election.cc:290] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:07.942835 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:07.949832 23063 raft_consensus.cc:513] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.950652 22557 ts_tablet_manager.cc:1918] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:07.950986 22557 log.cc:1198] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/1df6282224b849a488071323e2e69e1c
I20250112 08:57:07.951560 22557 ts_tablet_manager.cc:1939] T 1df6282224b849a488071323e2e69e1c P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:07.953416 22557 tablet_replica.cc:331] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:07.954097 22557 raft_consensus.cc:2238] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:07.954713 22557 pending_rounds.cc:62] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Trying to abort 1 pending ops.
I20250112 08:57:07.954908 22557 pending_rounds.cc:69] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409442895273984 op_type: NO_OP noop_request { }
I20250112 08:57:07.964004 22557 raft_consensus.cc:2883] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:07.964382 22557 raft_consensus.cc:2267] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:07.959884 22456 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:07.965078 22456 ts_tablet_manager.cc:1397] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.008s
I20250112 08:57:07.967347 22456 raft_consensus.cc:357] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.968014 22456 raft_consensus.cc:383] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:07.968242 22456 raft_consensus.cc:738] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:07.968829 22456 consensus_queue.cc:260] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:07.956518 22325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d730646b5b04e5eb9dd9cce1db90638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.973903 22557 ts_tablet_manager.cc:1905] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:07.975973 23063 leader_election.cc:290] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:07.979871 22456 ts_tablet_manager.cc:1428] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250112 08:57:07.980723 22456 tablet_bootstrap.cc:492] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:07.976732 22327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc909a01a6340dcb6902eeccf821f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:07.997144 22461 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:07.997713 22461 ts_tablet_manager.cc:1397] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.004s
I20250112 08:57:08.000283 22557 ts_tablet_manager.cc:1918] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:08.000604 22557 log.cc:1198] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/267ab3e5955b4d3bbeca44baf021b328
I20250112 08:57:08.001237 22557 ts_tablet_manager.cc:1939] T 267ab3e5955b4d3bbeca44baf021b328 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:08.003265 22557 tablet_replica.cc:331] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:08.005852 21709 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_cb9e1fb387114cfe86c64bcdc7a1c4c5" } new_table_name: "default.Table_6ba47fafc72c4861b7adf19ac9fba3cf" modify_external_catalogs: true
I20250112 08:57:08.008832 21712 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_ba481b6297ae48a4b20921031aef6276"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.010285 21712 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba481b6297ae48a4b20921031aef6276 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.018596 21716 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:08.019305 21716 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:08.020104 21716 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 7ad9b1593a24424989f380811954439e (table default.Table_95d4394b178c46188d58426a815a0964 [id=650e7fc35dcf4384a76a47c2d6f24665]). Expected version 2 got 0
I20250112 08:57:08.020224 22456 tablet_bootstrap.cc:654] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.021411 21716 catalog_manager.cc:5526] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:08.025053 21716 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:08.025930 21716 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
W20250112 08:57:08.026930 21704 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 267ab3e5955b4d3bbeca44baf021b328 consensus state has no leader
I20250112 08:57:08.028872 21710 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56650 (request call id 255) took 1767 ms. Trace:
I20250112 08:57:08.030000 22557 raft_consensus.cc:2238] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.030442 22557 raft_consensus.cc:2267] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.031937 22557 ts_tablet_manager.cc:1905] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:08.034684 21711 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 45) took 1695 ms (client timeout 1000 ms). Trace:
W20250112 08:57:08.035001 21711 rpcz_store.cc:269] 0112 08:57:06.338862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.339850 (+ 988us) service_pool.cc:225] Handling call
0112 08:57:08.034620 (+1694770us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5172,"reported_tablets":4}
I20250112 08:57:08.035974 23063 raft_consensus.cc:491] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.029284 21710 rpcz_store.cc:276] 0112 08:57:06.261109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.261204 (+ 95us) service_pool.cc:225] Handling call
0112 08:57:06.311655 (+ 50451us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:57:06.312368 (+ 713us) catalog_manager.cc:3627] Apply alter schema
0112 08:57:06.312888 (+ 520us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 08:57:06.313088 (+ 200us) catalog_manager.cc:3817] Updating metadata on disk
0112 08:57:06.411869 (+ 98781us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 08:57:08.028793 (+1616924us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:06.314282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:06.314635 (+ 353us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:06.314662 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:06.314674 (+ 12us) tablet.cc:659] Decoding operations
0112 08:57:06.314911 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:06.314948 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:06.314968 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:06.315095 (+ 127us) tablet.cc:698] Row locks acquired
0112 08:57:06.315107 (+ 12us) write_op.cc:260] PREPARE: finished
0112 08:57:06.315207 (+ 100us) write_op.cc:270] Start()
0112 08:57:06.315302 (+ 95us) write_op.cc:276] Timestamp: P: 1736672226315149 usec, L: 0
0112 08:57:06.315316 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:57:06.315740 (+ 424us) log.cc:844] Serialized 405 byte log entry
0112 08:57:06.401399 (+ 85659us) op_driver.cc:464] REPLICATION: finished
0112 08:57:06.409719 (+ 8320us) write_op.cc:301] APPLY: starting
0112 08:57:06.409813 (+ 94us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:06.409926 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:06.410028 (+ 102us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:06.410213 (+ 185us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:06.410357 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:06.410417 (+ 60us) write_op.cc:312] APPLY: finished
0112 08:57:06.410624 (+ 207us) log.cc:844] Serialized 23 byte log entry
0112 08:57:06.410782 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:06.410848 (+ 66us) write_op.cc:454] Released schema lock
0112 08:57:06.410887 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:06.411120 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:06.411151 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2388,"apply.run_cpu_time_us":1390,"apply.run_wall_time_us":7170,"num_ops":1,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":1711,"prepare.run_wall_time_us":1712,"raft.queue_time_us":1359,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":86011}]]}
I20250112 08:57:08.036429 23063 raft_consensus.cc:513] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.038220 23063 leader_election.cc:290] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:08.038774 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 010ce6f55c444fe0852ad72cc7383cc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 72e8f51e50994929b40a5e0679b07d8c) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:08.041034 22456 tablet_bootstrap.cc:492] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.041440 22456 ts_tablet_manager.cc:1397] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.004s
I20250112 08:57:08.043790 22456 raft_consensus.cc:357] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.044472 22456 raft_consensus.cc:383] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.044705 22456 raft_consensus.cc:738] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.045226 22456 consensus_queue.cc:260] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.048918 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_03c9aad903a742bb96dd314764735ea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.050855 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03c9aad903a742bb96dd314764735ea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.055035 21710 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e0510c7c9f7b4a39b4229444845528fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.056718 21710 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0510c7c9f7b4a39b4229444845528fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.057231 22456 ts_tablet_manager.cc:1428] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250112 08:57:08.058009 22456 tablet_bootstrap.cc:492] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.061165 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.066076 22456 tablet_bootstrap.cc:654] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.071589 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_2b22da2a0636476d9ba405517af8d41c" } modify_external_catalogs: true
I20250112 08:57:08.072180 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_2b22da2a0636476d9ba405517af8d41c" } modify_external_catalogs: true
W20250112 08:57:08.075430 22002 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 109767us
I20250112 08:57:08.076680 21715 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_9bf70ac0df104d21b87badcbe3f09bf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:08.077167 22557 ts_tablet_manager.cc:1918] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:08.077507 22557 log.cc:1198] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/5cd402674080464cb3bde3697ac23180
I20250112 08:57:08.078061 22557 ts_tablet_manager.cc:1939] T 5cd402674080464cb3bde3697ac23180 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:57:08.078053 21715 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bf70ac0df104d21b87badcbe3f09bf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.079187 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 422e1484cd84479abba824cc8a6fae1f
W20250112 08:57:08.082051 21708 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 42) took 1325 ms (client timeout 1000 ms). Trace:
I20250112 08:57:08.082353 22456 tablet_bootstrap.cc:492] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.082702 22456 ts_tablet_manager.cc:1397] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.001s
W20250112 08:57:08.082331 21708 rpcz_store.cc:269] 0112 08:57:06.757032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.757173 (+ 141us) service_pool.cc:225] Handling call
0112 08:57:08.081992 (+1324819us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:07.867177 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:07.870645 (+ 3468us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:07.870672 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:07.870685 (+ 13us) tablet.cc:659] Decoding operations
0112 08:57:07.871144 (+ 459us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:07.871181 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:07.871201 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:07.871344 (+ 143us) tablet.cc:698] Row locks acquired
0112 08:57:07.871356 (+ 12us) write_op.cc:260] PREPARE: finished
0112 08:57:07.871460 (+ 104us) write_op.cc:270] Start()
0112 08:57:07.871537 (+ 77us) write_op.cc:276] Timestamp: P: 1736672227871400 usec, L: 0
0112 08:57:07.871552 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 08:57:07.873070 (+ 1518us) log.cc:844] Serialized 533 byte log entry
0112 08:57:08.006911 (+133841us) op_driver.cc:464] REPLICATION: finished
0112 08:57:08.008069 (+ 1158us) write_op.cc:301] APPLY: starting
0112 08:57:08.008146 (+ 77us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:08.008255 (+ 109us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:08.008375 (+ 120us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:08.008584 (+ 209us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:08.008767 (+ 183us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:08.008827 (+ 60us) write_op.cc:312] APPLY: finished
0112 08:57:08.009030 (+ 203us) log.cc:844] Serialized 23 byte log entry
0112 08:57:08.009217 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:08.009293 (+ 76us) write_op.cc:454] Released schema lock
0112 08:57:08.009333 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:08.009624 (+ 291us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:08.009659 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":963,"apply.run_cpu_time_us":1515,"apply.run_wall_time_us":1515,"num_ops":1,"prepare.queue_time_us":8936,"prepare.run_cpu_time_us":1947,"prepare.run_wall_time_us":8132,"replication_time_us":135234}]]}
I20250112 08:57:08.085770 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:08.088263 21713 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_bbbd78fbf43744eb9ce8bcc320e9755a [id=d999ba318b994d1a9bf35f14a004d1bf] alter complete (version 2)
I20250112 08:57:08.088764 22456 raft_consensus.cc:357] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.089370 22456 raft_consensus.cc:383] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.089601 22456 raft_consensus.cc:738] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.090080 22456 consensus_queue.cc:260] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.095808 22456 ts_tablet_manager.cc:1428] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.013s user 0.001s sys 0.005s
I20250112 08:57:08.096601 22456 tablet_bootstrap.cc:492] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.101410 22456 tablet_bootstrap.cc:654] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:08.097401 22038 consensus_meta.cc:287] T cbeaee2ab95140f8825a6ca59c27701b P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 0.569s user 0.003s sys 0.004s
I20250112 08:57:08.107489 22326 tablet_service.cc:1467] Processing CreateTablet for tablet ab70dad5a1424df889ce4384292deb4b (DEFAULT_TABLE table=default.Table_0470958d2a10492bbc3bf701381a4095 [id=e0605f8bfdc64cefbe1bd2666f425a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.117671 22456 tablet_bootstrap.cc:492] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.118119 22456 ts_tablet_manager.cc:1397] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20250112 08:57:08.120194 22456 raft_consensus.cc:357] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:08.120772 22456 raft_consensus.cc:383] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.120668 22044 tablet_service.cc:1467] Processing CreateTablet for tablet ccc909a01a6340dcb6902eeccf821f5d (DEFAULT_TABLE table=default.Table_b648fe65974c47f085a8cf94713be26e [id=288f1c6fb89244709fec83bbe7507602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.121019 22456 raft_consensus.cc:738] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:57:08.120949 23041 log.cc:926] Time spent T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Append to log took a long time: real 0.270s user 0.002s sys 0.000s
I20250112 08:57:08.121712 22036 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:08.121510 22456 consensus_queue.cc:260] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:08.124372 22456 ts_tablet_manager.cc:1428] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:08.125088 22456 tablet_bootstrap.cc:492] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.129922 22456 tablet_bootstrap.cc:654] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.132287 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc909a01a6340dcb6902eeccf821f5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:08.123586 22290 consensus_peers.cc:487] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:08.134234 23063 consensus_queue.cc:1035] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:08.135632 22565 ts_tablet_manager.cc:1918] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:08.140009 22456 tablet_bootstrap.cc:492] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.140363 22456 ts_tablet_manager.cc:1397] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20250112 08:57:08.142727 22456 raft_consensus.cc:357] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.143389 22456 raft_consensus.cc:383] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.143622 22456 raft_consensus.cc:738] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.144158 22456 consensus_queue.cc:260] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.146360 22456 ts_tablet_manager.cc:1428] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:08.147099 22456 tablet_bootstrap.cc:492] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.149384 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:08.151868 22456 tablet_bootstrap.cc:654] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.157537 22456 tablet_bootstrap.cc:492] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.157915 22456 ts_tablet_manager.cc:1397] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20250112 08:57:08.159914 22456 raft_consensus.cc:357] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.160552 22456 raft_consensus.cc:383] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.160810 22456 raft_consensus.cc:738] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.161427 22456 consensus_queue.cc:260] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.163344 22326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab70dad5a1424df889ce4384292deb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.164001 22456 ts_tablet_manager.cc:1428] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 08:57:08.164773 22456 tablet_bootstrap.cc:492] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:57:08.162786 22037 consensus_meta.cc:287] T 1b7098f8f60445eb899434504ef59e4b P 76718d1596134c83900541c6d398d59e: Time spent flushing consensus metadata: real 0.630s user 0.004s sys 0.004s
I20250112 08:57:08.169543 22565 log.cc:1198] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/65591a0bb44148c49715b4564dd13524
I20250112 08:57:08.170276 22565 ts_tablet_manager.cc:1939] T 65591a0bb44148c49715b4564dd13524 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:08.170893 22456 tablet_bootstrap.cc:654] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.183668 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 592fc57d96cc42a7b3d07307bd39a557 (DEFAULT_TABLE table=default.Table_713a879ad53d408b90cdfad8f6d01a5b [id=0bb1db84ec2b41f5b40a81070d309799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.184772 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fc57d96cc42a7b3d07307bd39a557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.000185 22461 raft_consensus.cc:357] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.188060 22461 raft_consensus.cc:383] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.188330 22461 raft_consensus.cc:738] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:57:08.191723 21709 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1722022567174f7a8cc9c85fe6655b4c has no consensus state
I20250112 08:57:08.197469 22456 tablet_bootstrap.cc:492] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.197852 22456 ts_tablet_manager.cc:1397] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.004s
I20250112 08:57:08.199927 22456 raft_consensus.cc:357] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:08.200569 22456 raft_consensus.cc:383] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.200799 22456 raft_consensus.cc:738] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.201263 22456 consensus_queue.cc:260] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:57:08.204814 23107 log.cc:926] Time spent T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250112 08:57:08.200932 22461 consensus_queue.cc:260] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.213129 22461 ts_tablet_manager.cc:1428] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.215s user 0.005s sys 0.000s
I20250112 08:57:08.213868 22461 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:08.219389 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.220191 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 130) took 5286 ms. Trace:
I20250112 08:57:08.220496 22565 rpcz_store.cc:276] 0112 08:57:02.934047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.934172 (+ 125us) service_pool.cc:225] Handling call
0112 08:57:07.461170 (+4526998us) ts_tablet_manager.cc:1145] Deleting tablet 65591a0bb44148c49715b4564dd13524
0112 08:57:07.461247 (+ 77us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:07.745998 (+284751us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.135408 (+389410us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.220141 (+ 84733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4481188}
I20250112 08:57:08.221583 22565 tablet_replica.cc:331] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:08.222458 22565 raft_consensus.cc:2238] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.222864 22565 raft_consensus.cc:2267] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.224615 22565 ts_tablet_manager.cc:1905] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:08.225355 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:08.229200 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b38deeff1964b09938a12cfd4992ec5" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.235054 22461 tablet_bootstrap.cc:654] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.237982 23103 ts_tablet_manager.cc:1918] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:08.238407 23103 log.cc:1198] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/267ab3e5955b4d3bbeca44baf021b328
W20250112 08:57:08.238646 21999 leader_election.cc:343] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:08.239159 23103 ts_tablet_manager.cc:1939] T 267ab3e5955b4d3bbeca44baf021b328 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
W20250112 08:57:08.245645 22290 leader_election.cc:343] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.249562 22290 leader_election.cc:343] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.252092 23041 log.cc:926] Time spent T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: Append to log took a long time: real 0.076s user 0.003s sys 0.000s
I20250112 08:57:08.252458 23063 consensus_queue.cc:1035] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:08.253706 22290 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.254587 22290 consensus_peers.cc:487] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:08.268770 21714 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet c4fc4d2d0eaa47f6a02f518828253057
I20250112 08:57:08.269381 21712 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_00d2c67bdfbd47a6a03b2fb22f72f193" } new_table_name: "default.Table_eb6046dddf9e4a28a8254e74a7cce16a" modify_external_catalogs: true
I20250112 08:57:08.274407 22325 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:08.275085 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
I20250112 08:57:08.277325 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 65591a0bb44148c49715b4564dd13524 (table default.Table_ca0708b2b934445fbd13bbf81e61f61f [id=f40624aa29a1492aa24df3a322a4b5d7]) successfully deleted
I20250112 08:57:08.296417 22181 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.296289 22184 tablet_service.cc:1467] Processing CreateTablet for tablet ab70dad5a1424df889ce4384292deb4b (DEFAULT_TABLE table=default.Table_0470958d2a10492bbc3bf701381a4095 [id=e0605f8bfdc64cefbe1bd2666f425a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.299095 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab70dad5a1424df889ce4384292deb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.302708 22186 tablet_service.cc:1467] Processing CreateTablet for tablet ccc909a01a6340dcb6902eeccf821f5d (DEFAULT_TABLE table=default.Table_b648fe65974c47f085a8cf94713be26e [id=288f1c6fb89244709fec83bbe7507602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.304275 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.305106 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.306034 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet ea15d9405b9944db8e00121965b9ffde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.306994 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e62435dce1441a3b2fee50518662d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:00 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.308117 21999 leader_election.cc:304] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:08.308809 21999 raft_consensus.cc:2681] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 08:57:08.309271 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 70674.2us
I20250112 08:57:08.265606 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 7d730646b5b04e5eb9dd9cce1db90638 (DEFAULT_TABLE table=default.Table_c1a22dbcea954ac1884b4f750e841968 [id=adcc38f793b34989b3369c9d5e143295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.283391 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.310164 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.310900 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d730646b5b04e5eb9dd9cce1db90638. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.283974 22043 tablet_service.cc:1467] Processing CreateTablet for tablet c2f340b047cd4112a56b0482a90c1db3 (DEFAULT_TABLE table=default.Table_fe10565e765240e9b7dd43a3f4d2e6a6 [id=8022b1aa8763462f90039c83a1122228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.295039 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 592fc57d96cc42a7b3d07307bd39a557 (DEFAULT_TABLE table=default.Table_713a879ad53d408b90cdfad8f6d01a5b [id=0bb1db84ec2b41f5b40a81070d309799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.312157 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fc57d96cc42a7b3d07307bd39a557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.298357 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:08.304538 23105 raft_consensus.cc:491] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.320089 23105 raft_consensus.cc:513] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.296291 22183 tablet_service.cc:1467] Processing CreateTablet for tablet 592fc57d96cc42a7b3d07307bd39a557 (DEFAULT_TABLE table=default.Table_713a879ad53d408b90cdfad8f6d01a5b [id=0bb1db84ec2b41f5b40a81070d309799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.321870 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccc909a01a6340dcb6902eeccf821f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.322073 23105 leader_election.cc:290] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:08.324563 22002 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: 267ab3e5955b4d3bbeca44baf021b328
I20250112 08:57:08.328550 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f340b047cd4112a56b0482a90c1db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.335069 22458 tablet_bootstrap.cc:492] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:08.335467 22458 ts_tablet_manager.cc:1397] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.884s user 0.011s sys 0.008s
I20250112 08:57:08.337380 22456 ts_tablet_manager.cc:1428] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.139s user 0.005s sys 0.000s
W20250112 08:57:08.338442 22146 leader_election.cc:343] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.339033 22146 leader_election.cc:304] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:08.344857 22183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 592fc57d96cc42a7b3d07307bd39a557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.345360 22456 tablet_bootstrap.cc:492] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.351382 22456 tablet_bootstrap.cc:654] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.351169 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:08.358181 22212 raft_consensus.cc:3054] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:08.360756 22039 tablet_service.cc:1467] Processing CreateTablet for tablet ab70dad5a1424df889ce4384292deb4b (DEFAULT_TABLE table=default.Table_0470958d2a10492bbc3bf701381a4095 [id=e0605f8bfdc64cefbe1bd2666f425a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:08.362141 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab70dad5a1424df889ce4384292deb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:08.362943 23009 raft_consensus.cc:2743] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:08.381441 22290 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:08.381882 22290 leader_election.cc:304] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:08.382938 23099 raft_consensus.cc:2743] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:08.384701 23070 consensus_queue.cc:1035] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:08.385866 22461 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:08.386354 22461 ts_tablet_manager.cc:1397] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.173s user 0.006s sys 0.011s
I20250112 08:57:08.388743 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 1df6282224b849a488071323e2e69e1c (table default.Table_3bb2b43e854340fba2b4c4ad8a3da14a [id=eea0053edb8c4c8f8d0bd48fd076a0c5]) successfully deleted
I20250112 08:57:08.389279 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
I20250112 08:57:08.389212 21712 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ee92de1efccf428da7c2c5e6204cd4b5" } modify_external_catalogs: true
I20250112 08:57:08.389786 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 5cd402674080464cb3bde3697ac23180 (table default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059]) successfully deleted
I20250112 08:57:08.389724 21712 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_ee92de1efccf428da7c2c5e6204cd4b5" } modify_external_catalogs: true
W20250112 08:57:08.390331 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 already in progress: creating tablet
I20250112 08:57:08.402364 22557 tablet_replica.cc:331] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:08.367769 22458 raft_consensus.cc:357] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.404138 22458 raft_consensus.cc:383] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.404438 22458 raft_consensus.cc:738] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.405232 22458 consensus_queue.cc:260] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.408869 22456 tablet_bootstrap.cc:492] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.409268 22456 ts_tablet_manager.cc:1397] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.005s
W20250112 08:57:08.413651 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet c4fc4d2d0eaa47f6a02f518828253057 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4fc4d2d0eaa47f6a02f518828253057 already in progress: creating tablet
I20250112 08:57:08.417794 23126 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:08.418251 23126 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:08.427129 23078 catalog_manager.cc:1260] Loaded cluster ID: 99ee22f2d13d4f5080e046f96817436c
I20250112 08:57:08.427392 23078 catalog_manager.cc:1553] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: loading cluster ID for follower catalog manager: success
I20250112 08:57:08.433156 23126 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:08.434824 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:08.435492 21712 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 1ad1f85596894fe9992ec4ad115e6928
I20250112 08:57:08.440230 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:08.443269 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
W20250112 08:57:08.446593 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 60991.9us
I20250112 08:57:08.447799 22461 raft_consensus.cc:357] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.448475 22461 raft_consensus.cc:383] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.448712 22461 raft_consensus.cc:738] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.449206 22461 consensus_queue.cc:260] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.451346 22461 ts_tablet_manager.cc:1428] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20250112 08:57:08.452167 22461 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:08.453467 22565 ts_tablet_manager.cc:1918] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:08.453877 22565 log.cc:1198] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/dabc37f08d1948118b98ba25b689bea4
I20250112 08:57:08.454557 22565 ts_tablet_manager.cc:1939] T dabc37f08d1948118b98ba25b689bea4 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:08.446699 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:08.414359 22212 raft_consensus.cc:2463] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:08.461097 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:08.356535 23142 raft_consensus.cc:491] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:57:08.357581 22146 leader_election.cc:343] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.464336 22149 leader_election.cc:343] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.464205 23142 raft_consensus.cc:513] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.469018 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:08.470234 22290 leader_election.cc:304] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:08.471258 23070 raft_consensus.cc:2798] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:08.471979 23070 raft_consensus.cc:695] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:08.472806 23070 consensus_queue.cc:237] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.476410 23103 tablet_replica.cc:331] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:08.478291 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_b2e28d9ccf214a9aafc78b9f7e74328b" } new_table_name: "default.Table_60d957dcf38841e5a1ab61b409df19af" modify_external_catalogs: true
I20250112 08:57:08.477432 23103 raft_consensus.cc:2238] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.487074 23103 raft_consensus.cc:2267] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.493417 22458 ts_tablet_manager.cc:1428] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.158s user 0.007s sys 0.000s
I20250112 08:57:08.494335 22458 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:08.468245 22146 leader_election.cc:304] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:08.508437 23143 raft_consensus.cc:2743] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:08.516139 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 131) took 5581 ms. Trace:
I20250112 08:57:08.516495 22565 rpcz_store.cc:276] 0112 08:57:02.935220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.935369 (+ 149us) service_pool.cc:225] Handling call
0112 08:57:08.221398 (+5286029us) ts_tablet_manager.cc:1145] Deleting tablet dabc37f08d1948118b98ba25b689bea4
0112 08:57:08.221482 (+ 84us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:08.433294 (+211812us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.453215 (+ 19921us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.516074 (+ 62859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5241349}
W20250112 08:57:08.516645 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 131742us
I20250112 08:57:08.520247 23143 consensus_queue.cc:1035] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250112 08:57:08.523564 22146 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 166014us
I20250112 08:57:08.525684 23143 raft_consensus.cc:491] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.534583 23136 raft_consensus.cc:491] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c608785d6874abf9cc28db11a979ddd)
I20250112 08:57:08.535036 23136 raft_consensus.cc:513] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.536881 23136 leader_election.cc:290] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:08.543211 22565 tablet_replica.cc:331] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:08.546296 22063 raft_consensus.cc:1270] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Refusing update from remote peer 53068c99960647728141706e9c50b870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:57:08.551877 22044 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad1f85596894fe9992ec4ad115e6928 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:08.554610 22458 tablet_bootstrap.cc:654] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.579161 23136 consensus_queue.cc:1035] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:08.526086 23143 raft_consensus.cc:513] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.581574 23143 leader_election.cc:290] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:08.411826 22456 raft_consensus.cc:357] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.586683 23142 leader_election.cc:290] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:08.588358 22565 raft_consensus.cc:2238] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.588775 22565 pending_rounds.cc:62] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Trying to abort 1 pending ops.
I20250112 08:57:08.588959 22565 pending_rounds.cc:69] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409428913528832 op_type: NO_OP noop_request { }
I20250112 08:57:08.589818 22565 raft_consensus.cc:2883] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:08.590153 22565 raft_consensus.cc:2267] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.590730 23037 consensus_queue.cc:1035] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:57:08.586762 23047 raft_consensus.cc:491] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.592877 23047 raft_consensus.cc:513] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.595075 23047 leader_election.cc:290] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:08.597563 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46707de727a49108afb12ea2cf08aa0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:08.598066 21999 leader_election.cc:343] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 267ab3e5955b4d3bbeca44baf021b328
I20250112 08:57:08.598485 21999 leader_election.cc:304] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:08.598587 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e8f51e50994929b40a5e0679b07d8c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.599258 23096 raft_consensus.cc:2743] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:08.599731 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:08.600690 22461 tablet_bootstrap.cc:654] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:08.601810 22146 leader_election.cc:343] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.602536 22146 leader_election.cc:343] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:08.603195 22146 leader_election.cc:343] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.603976 22456 raft_consensus.cc:383] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.604269 22456 raft_consensus.cc:738] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.605683 22557 raft_consensus.cc:2238] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.606251 22557 raft_consensus.cc:2267] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.608835 22557 ts_tablet_manager.cc:1905] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:08.604965 22456 consensus_queue.cc:260] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.617087 22456 ts_tablet_manager.cc:1428] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.208s user 0.007s sys 0.000s
I20250112 08:57:08.618052 22456 tablet_bootstrap.cc:492] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:08.634423 22565 ts_tablet_manager.cc:1905] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:08.644732 21708 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_8597641f7fe6456fb720caf6be070864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:08.647429 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet dabc37f08d1948118b98ba25b689bea4 (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]) successfully deleted
W20250112 08:57:08.649262 21708 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8597641f7fe6456fb720caf6be070864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:08.649578 21999 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:08.654405 21709 master_service.cc:946] ReplaceTablet: received request to replace tablet 010ce6f55c444fe0852ad72cc7383cc1 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:08.664709 21711 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_e24a40491b8f455eb04b0a4e567df5d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.666213 21711 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e24a40491b8f455eb04b0a4e567df5d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.670996 22350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e8f51e50994929b40a5e0679b07d8c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:08.672210 23103 ts_tablet_manager.cc:1905] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:08.674198 23078 catalog_manager.cc:1575] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: acquiring CA information for follower catalog manager: success
I20250112 08:57:08.693244 22458 tablet_bootstrap.cc:492] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:08.695400 22458 ts_tablet_manager.cc:1397] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.201s user 0.008s sys 0.011s
I20250112 08:57:08.697829 22458 raft_consensus.cc:357] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.698696 22458 raft_consensus.cc:383] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.698997 22458 raft_consensus.cc:738] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.703078 22458 consensus_queue.cc:260] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.712908 22458 ts_tablet_manager.cc:1428] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250112 08:57:08.713922 22458 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:08.714078 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46707de727a49108afb12ea2cf08aa0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:08.718098 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:08.719434 22458 tablet_bootstrap.cc:654] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.702776 23037 raft_consensus.cc:491] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.709496 22565 ts_tablet_manager.cc:1918] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 08:57:08.740257 21999 consensus_peers.cc:487] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06717cb567a54ee6a268c3be5d8786ec. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:08.737289 22565 log.cc:1198] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:08.741767 22565 ts_tablet_manager.cc:1939] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:08.735599 23037 raft_consensus.cc:513] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.745831 23037 leader_election.cc:290] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:08.747223 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6368a80a9fe9457a803e68a2392b638d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:08.752251 22146 leader_election.cc:343] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.757865 21713 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f3931a3aec8748a2987f29762a28d078"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.759443 21713 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3931a3aec8748a2987f29762a28d078 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.760586 21714 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_a6bd091b79fd4a2c952ed2f0f95da386" } modify_external_catalogs: true
I20250112 08:57:08.766875 22456 tablet_bootstrap.cc:654] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.767364 23161 tablet.cc:1721] T a5959caa4d39420981e3b1994df4cb17 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:08.763870 23167 tablet.cc:1721] T a5959caa4d39420981e3b1994df4cb17 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:08.781330 22461 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:08.781816 22461 ts_tablet_manager.cc:1397] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.330s user 0.006s sys 0.006s
I20250112 08:57:08.784190 22461 raft_consensus.cc:357] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.785795 23078 catalog_manager.cc:1603] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:08.785974 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_214464b532cf4fc997136cca2adb554a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:08.787499 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_214464b532cf4fc997136cca2adb554a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:08.764791 22557 ts_tablet_manager.cc:1918] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:08.797744 22557 log.cc:1198] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/422e1484cd84479abba824cc8a6fae1f
I20250112 08:57:08.818575 22461 raft_consensus.cc:383] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.818926 22461 raft_consensus.cc:738] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.821126 22461 consensus_queue.cc:260] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.823189 22461 ts_tablet_manager.cc:1428] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250112 08:57:08.824067 22461 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:08.765915 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 132) took 5830 ms. Trace:
I20250112 08:57:08.825131 22565 rpcz_store.cc:276] 0112 08:57:02.935539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:02.972022 (+ 36483us) service_pool.cc:225] Handling call
0112 08:57:08.517349 (+5545327us) ts_tablet_manager.cc:1145] Deleting tablet 14c7056aaf8844da915acec5f9f92eeb
0112 08:57:08.543074 (+ 25725us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:08.665914 (+122840us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.709267 (+ 43353us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.765858 (+ 56591us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11001,"tablet-delete.queue_time_us":5530145}
I20250112 08:57:08.827921 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 14c7056aaf8844da915acec5f9f92eeb (table default.Table_3649588cf8414d9591e8f2ec3fb78e44 [id=90bd60a763ae479f8ac8122d75301bab]) successfully deleted
I20250112 08:57:08.829335 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 137) took 5768 ms. Trace:
I20250112 08:57:08.832276 22565 rpcz_store.cc:276] 0112 08:57:03.061374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.071018 (+ 9644us) service_pool.cc:225] Handling call
0112 08:57:08.828958 (+5757940us) ts_tablet_manager.cc:1145] Deleting tablet 92a22f87fbbf48e88dbd5af418ee94ad
0112 08:57:08.829006 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:08.829269 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757313}
W20250112 08:57:08.833673 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 92a22f87fbbf48e88dbd5af418ee94ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92a22f87fbbf48e88dbd5af418ee94ad already in progress: creating tablet
I20250112 08:57:08.834825 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 141) took 5745 ms. Trace:
I20250112 08:57:08.838730 22557 ts_tablet_manager.cc:1939] T 422e1484cd84479abba824cc8a6fae1f P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:08.839066 22565 rpcz_store.cc:276] 0112 08:57:03.089780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.089915 (+ 135us) service_pool.cc:225] Handling call
0112 08:57:08.834478 (+5744563us) ts_tablet_manager.cc:1145] Deleting tablet 6c34fc4b6aaf45fdae76cf153129f2da
0112 08:57:08.834530 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:08.834777 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5744021}
I20250112 08:57:08.841243 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 422e1484cd84479abba824cc8a6fae1f (table default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]) successfully deleted
W20250112 08:57:08.842221 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 6c34fc4b6aaf45fdae76cf153129f2da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c34fc4b6aaf45fdae76cf153129f2da already in progress: creating tablet
I20250112 08:57:08.843230 22565 tablet_replica.cc:331] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:08.844055 23169 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:08.844051 22565 raft_consensus.cc:2238] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:08.845440 22565 raft_consensus.cc:2267] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:08.847337 22565 ts_tablet_manager.cc:1905] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:08.848181 23142 raft_consensus.cc:491] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:08.848620 23142 raft_consensus.cc:513] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:08.850550 23142 leader_election.cc:290] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:08.852888 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c34fc4b6aaf45fdae76cf153129f2da" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:08.855105 22146 leader_election.cc:343] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.869149 22461 tablet_bootstrap.cc:654] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:08.870117 22456 tablet_bootstrap.cc:492] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:08.870563 22456 ts_tablet_manager.cc:1397] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.253s user 0.008s sys 0.002s
I20250112 08:57:08.873256 22456 raft_consensus.cc:357] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.873924 22456 raft_consensus.cc:383] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:08.874177 22456 raft_consensus.cc:738] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.874768 22456 consensus_queue.cc:260] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.915431 23161 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:08.924116 22456 ts_tablet_manager.cc:1428] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
W20250112 08:57:08.948959 22149 leader_election.cc:343] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.949481 22149 leader_election.cc:304] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:08.950728 23143 raft_consensus.cc:2743] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:08.955320 22149 leader_election.cc:343] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.955700 22149 leader_election.cc:304] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:08.956697 23165 raft_consensus.cc:2743] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:08.966087 22565 ts_tablet_manager.cc:1918] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:08.967962 22461 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:08.968338 22461 ts_tablet_manager.cc:1397] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.024s
I20250112 08:57:08.970490 22461 raft_consensus.cc:357] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.971122 22461 raft_consensus.cc:383] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 08:57:08.982717 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 137e2275af51421cac4ca629dfa809ba has no consensus state
I20250112 08:57:08.971362 22461 raft_consensus.cc:738] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:08.983355 22461 consensus_queue.cc:260] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:08.984772 21710 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_538457c36d5643a2a4f3a40ebbbb25f9" } new_table_name: "default.Table_b576158684104afd8673ccbe72b45c40" modify_external_catalogs: true
I20250112 08:57:08.989176 21708 master_service.cc:946] ReplaceTablet: received request to replace tablet 14c7056aaf8844da915acec5f9f92eeb from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:08.986264 22461 ts_tablet_manager.cc:1428] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 08:57:08.992103 21708 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_55a599c370f441969aad1da2ebfb4793" } modify_external_catalogs: true
I20250112 08:57:08.992682 22461 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:08.998575 22149 leader_election.cc:343] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:08.999287 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:08.999830 22149 leader_election.cc:304] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.000676 23165 raft_consensus.cc:2743] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:09.004200 22557 tablet_replica.cc:331] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:09.005077 22557 raft_consensus.cc:2238] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:09.005595 22565 log.cc:1198] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:09.006866 22565 ts_tablet_manager.cc:1939] T 010ce6f55c444fe0852ad72cc7383cc1 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:09.007320 22461 tablet_bootstrap.cc:654] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.009689 22456 tablet_bootstrap.cc:492] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:09.015053 22456 tablet_bootstrap.cc:654] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.018167 22557 raft_consensus.cc:2267] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:09.021085 22557 ts_tablet_manager.cc:1905] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:09.025094 22461 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:09.025475 22461 ts_tablet_manager.cc:1397] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.000s
I20250112 08:57:09.028033 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 145) took 5937 ms. Trace:
I20250112 08:57:09.027643 22461 raft_consensus.cc:357] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.029004 22461 raft_consensus.cc:383] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 08:57:09.029098 22002 leader_election.cc:343] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: 14c7056aaf8844da915acec5f9f92eeb
I20250112 08:57:09.029263 22461 raft_consensus.cc:738] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.029470 22002 leader_election.cc:304] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:09.029973 22002 raft_consensus.cc:2681] T 14c7056aaf8844da915acec5f9f92eeb P 76718d1596134c83900541c6d398d59e: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:57:09.029799 22461 consensus_queue.cc:260] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.033440 22461 ts_tablet_manager.cc:1428] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 08:57:09.034220 22461 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:09.034334 22458 tablet_bootstrap.cc:492] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:09.034811 22458 ts_tablet_manager.cc:1397] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.321s user 0.013s sys 0.013s
I20250112 08:57:09.037004 22458 raft_consensus.cc:357] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.037712 22458 raft_consensus.cc:383] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.038021 22458 raft_consensus.cc:738] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.028422 22565 rpcz_store.cc:276] 0112 08:57:03.090947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.092832 (+ 1885us) service_pool.cc:225] Handling call
0112 08:57:08.843034 (+5750202us) ts_tablet_manager.cc:1145] Deleting tablet 010ce6f55c444fe0852ad72cc7383cc1
0112 08:57:08.843088 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:08.908905 (+ 65817us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:08.965877 (+ 56972us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.027974 (+ 62097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5749722}
I20250112 08:57:09.041550 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 146) took 5950 ms. Trace:
I20250112 08:57:09.040417 23131 consensus_queue.cc:1035] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:09.038978 22458 consensus_queue.cc:260] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.049005 22565 rpcz_store.cc:276] 0112 08:57:03.091224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.093432 (+ 2208us) service_pool.cc:225] Handling call
0112 08:57:09.041191 (+5947759us) ts_tablet_manager.cc:1145] Deleting tablet fc82026fcd1541418176a015ea3253cf
0112 08:57:09.041238 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.041501 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5947182}
I20250112 08:57:09.051752 23096 raft_consensus.cc:491] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.060835 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 147) took 5969 ms. Trace:
I20250112 08:57:09.060761 23096 raft_consensus.cc:513] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.061141 22565 rpcz_store.cc:276] 0112 08:57:03.091514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.094068 (+ 2554us) service_pool.cc:225] Handling call
0112 08:57:09.060450 (+5966382us) ts_tablet_manager.cc:1145] Deleting tablet f9835288a7974e85821b095c03e91886
0112 08:57:09.060501 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.060782 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5965845}
I20250112 08:57:09.064299 22456 tablet_bootstrap.cc:492] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:09.064725 22456 ts_tablet_manager.cc:1397] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.001s
I20250112 08:57:09.065452 22461 tablet_bootstrap.cc:654] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.066654 23096 leader_election.cc:290] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.066864 22456 raft_consensus.cc:357] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.067485 22456 raft_consensus.cc:383] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.055590 22458 ts_tablet_manager.cc:1428] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20250112 08:57:09.067553 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:09.068344 22209 raft_consensus.cc:2388] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:09.071136 23165 raft_consensus.cc:491] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.071583 23165 raft_consensus.cc:513] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:09.073650 23165 leader_election.cc:290] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:09.075567 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e924f523524bd0a387597399e8470a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:09.076874 22146 leader_election.cc:343] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.080086 22458 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:09.082006 22461 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:09.067783 22456 raft_consensus.cc:738] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.082398 22461 ts_tablet_manager.cc:1397] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.004s
I20250112 08:57:09.084725 22461 raft_consensus.cc:357] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.085407 22461 raft_consensus.cc:383] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.085661 22461 raft_consensus.cc:738] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.085216 22456 consensus_queue.cc:260] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.086355 22461 consensus_queue.cc:260] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.092553 22565 tablet_replica.cc:331] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:09.097077 22456 ts_tablet_manager.cc:1428] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250112 08:57:09.097947 22456 tablet_bootstrap.cc:492] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:09.099031 22461 ts_tablet_manager.cc:1428] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250112 08:57:09.099980 22461 tablet_bootstrap.cc:492] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:09.104825 22565 raft_consensus.cc:2238] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:09.105540 22565 pending_rounds.cc:62] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Trying to abort 1 pending ops.
I20250112 08:57:09.105815 22565 pending_rounds.cc:69] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409444317483008 op_type: NO_OP noop_request { }
I20250112 08:57:09.106225 22565 raft_consensus.cc:2883] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:09.106592 22565 raft_consensus.cc:2267] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:09.109169 22565 ts_tablet_manager.cc:1905] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:09.113754 22458 tablet_bootstrap.cc:654] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.121008 23165 raft_consensus.cc:491] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.121491 23165 raft_consensus.cc:513] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.122408 22461 tablet_bootstrap.cc:654] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.123397 23165 leader_election.cc:290] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.124989 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:09.126727 22146 leader_election.cc:343] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.138816 23165 raft_consensus.cc:491] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.139282 23165 raft_consensus.cc:513] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.142529 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62bae16ab43414c8d8be2766802d4c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:09.143847 22146 leader_election.cc:343] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.144425 23165 leader_election.cc:290] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.150302 23096 raft_consensus.cc:491] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.150843 23096 raft_consensus.cc:513] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.154009 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:09.166208 22461 tablet_bootstrap.cc:492] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:09.166611 22461 ts_tablet_manager.cc:1397] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.008s
I20250112 08:57:09.167415 22456 tablet_bootstrap.cc:654] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.168931 22461 raft_consensus.cc:357] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.169669 22461 raft_consensus.cc:383] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.169937 22461 raft_consensus.cc:738] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.170557 22461 consensus_queue.cc:260] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.174727 23096 leader_election.cc:290] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.186730 22461 ts_tablet_manager.cc:1428] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250112 08:57:09.187296 21708 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_55a599c370f441969aad1da2ebfb4793" } modify_external_catalogs: true
I20250112 08:57:09.198457 22458 tablet_bootstrap.cc:492] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:09.199792 22458 ts_tablet_manager.cc:1397] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.120s user 0.006s sys 0.003s
I20250112 08:57:09.207975 22557 ts_tablet_manager.cc:1918] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:09.211525 22557 log.cc:1198] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/92a22f87fbbf48e88dbd5af418ee94ad
I20250112 08:57:09.213251 22557 ts_tablet_manager.cc:1939] T 92a22f87fbbf48e88dbd5af418ee94ad P 53068c99960647728141706e9c50b870: Deleting consensus metadata
W20250112 08:57:08.996865 21716 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 46) took 1756 ms (client timeout 1000 ms). Trace:
I20250112 08:57:09.046595 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 010ce6f55c444fe0852ad72cc7383cc1 (table default.Table_1b6fafd4ef8b49beba9040f711b5dac3 [id=64cfa083d78342629755cef4afbe7b61]) successfully deleted
W20250112 08:57:09.216763 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 170196us
W20250112 08:57:09.089515 21702 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 137e2275af51421cac4ca629dfa809ba (table default.Table_60d957dcf38841e5a1ab61b409df19af [id=3390616d770745a39ee27f66c84ce788], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 137e2275af51421cac4ca629dfa809ba has no consensus state
I20250112 08:57:09.170682 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
W20250112 08:57:09.180474 21710 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c228f04faa5c4f68a95adfb295235047 consensus state has no leader
I20250112 08:57:09.180917 21711 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:08.995450 21714 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_a6bd091b79fd4a2c952ed2f0f95da386" } modify_external_catalogs: true
I20250112 08:57:09.226477 22456 tablet_bootstrap.cc:492] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:09.226891 22456 ts_tablet_manager.cc:1397] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.000s
I20250112 08:57:09.240859 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:09.243700 23165 raft_consensus.cc:491] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.244324 23165 raft_consensus.cc:513] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.249356 23165 leader_election.cc:290] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.250053 22458 raft_consensus.cc:357] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.250924 22458 raft_consensus.cc:383] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.251267 22458 raft_consensus.cc:738] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.263623 22565 ts_tablet_manager.cc:1918] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:09.278157 23099 raft_consensus.cc:491] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.278312 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:09.278658 23099 raft_consensus.cc:513] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.279393 21713 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "default.Table_15a31e5fe0e74322bcbefd51437a08b7" } new_table_name: "default.Table_f6b0db0e8c6a4dea88649d3dc4286a81" modify_external_catalogs: true
I20250112 08:57:09.280591 23099 leader_election.cc:290] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:09.221349 21716 rpcz_store.cc:269] 0112 08:57:07.240452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.262021 (+ 21569us) service_pool.cc:225] Handling call
0112 08:57:08.996682 (+1734661us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:08.237703 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:08.238039 (+ 336us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:08.238061 (+ 22us) write_op.cc:435] Acquired schema lock
0112 08:57:08.238076 (+ 15us) tablet.cc:659] Decoding operations
0112 08:57:08.238324 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:08.238357 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:08.238383 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:08.238549 (+ 166us) tablet.cc:698] Row locks acquired
0112 08:57:08.238563 (+ 14us) write_op.cc:260] PREPARE: finished
0112 08:57:08.238658 (+ 95us) write_op.cc:270] Start()
0112 08:57:08.238728 (+ 70us) write_op.cc:276] Timestamp: P: 1736672228238611 usec, L: 0
0112 08:57:08.238743 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 08:57:08.239212 (+ 469us) log.cc:844] Serialized 533 byte log entry
0112 08:57:08.318482 (+ 79270us) op_driver.cc:464] REPLICATION: finished
0112 08:57:08.320422 (+ 1940us) write_op.cc:301] APPLY: starting
0112 08:57:08.320459 (+ 37us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:08.320532 (+ 73us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:08.320599 (+ 67us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:08.320757 (+ 158us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:08.320867 (+ 110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:08.320929 (+ 62us) write_op.cc:312] APPLY: finished
0112 08:57:08.321075 (+ 146us) log.cc:844] Serialized 23 byte log entry
0112 08:57:08.321185 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:08.321238 (+ 53us) write_op.cc:454] Released schema lock
0112 08:57:08.321266 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":49698,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":1830,"apply.run_cpu_time_us":1003,"apply.run_wall_time_us":1003,"num_ops":1,"prepare.queue_time_us":139318,"prepare.run_cpu_time_us":1742,"prepare.run_wall_time_us":14455,"replication_time_us":79657}]]}
I20250112 08:57:09.286018 21717 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "default.Table_f3352b4abff244e88e533d442b24437d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:09.328764 22461 tablet_bootstrap.cc:492] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:07.689786 22876 rpcz_store.cc:276] 0112 08:57:03.603491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.603666 (+ 175us) service_pool.cc:225] Handling call
0112 08:57:03.792813 (+189147us) raft_consensus.cc:1466] Updating replica for 216 ops
0112 08:57:03.824855 (+ 32042us) raft_consensus.cc:1529] Early marking committed up to index 10
0112 08:57:03.868602 (+ 43747us) raft_consensus.cc:1534] Triggering prepare for 214 ops
0112 08:57:03.869070 (+ 468us) write_op.cc:270] Start()
0112 08:57:03.869274 (+ 204us) write_op.cc:276] Timestamp: P: 1736672211365876 usec, L: 0
0112 08:57:03.891805 (+ 22531us) write_op.cc:270] Start()
0112 08:57:03.891934 (+ 129us) write_op.cc:276] Timestamp: P: 1736672211373148 usec, L: 0
0112 08:57:03.893953 (+ 2019us) write_op.cc:270] Start()
0112 08:57:03.894030 (+ 77us) write_op.cc:276] Timestamp: P: 1736672211381129 usec, L: 0
0112 08:57:03.896679 (+ 2649us) write_op.cc:270] Start()
0112 08:57:03.896763 (+ 84us) write_op.cc:276] Timestamp: P: 1736672211414898 usec, L: 0
0112 08:57:03.897286 (+ 523us) write_op.cc:270] Start()
0112 08:57:03.897375 (+ 89us) write_op.cc:276] Timestamp: P: 1736672211515771 usec, L: 0
0112 08:57:03.897804 (+ 429us) write_op.cc:270] Start()
0112 08:57:03.897898 (+ 94us) write_op.cc:276] Timestamp: P: 1736672211554631 usec, L: 0
0112 08:57:03.898330 (+ 432us) write_op.cc:270] Start()
0112 08:57:03.898499 (+ 169us) write_op.cc:276] Timestamp: P: 1736672211569342 usec, L: 0
0112 08:57:03.898968 (+ 469us) write_op.cc:270] Start()
0112 08:57:03.899049 (+ 81us) write_op.cc:276] Timestamp: P: 1736672211575012 usec, L: 0
0112 08:57:03.905197 (+ 6148us) write_op.cc:270] Start()
0112 08:57:03.905288 (+ 91us) write_op.cc:276] Timestamp: P: 1736672211595182 usec, L: 0
0112 08:57:03.905817 (+ 529us) write_op.cc:270] Start()
0112 08:57:03.905889 (+ 72us) write_op.cc:276] Timestamp: P: 1736672211676237 usec, L: 0
0112 08:57:03.906333 (+ 444us) write_op.cc:270] Start()
0112 08:57:03.906409 (+ 76us) write_op.cc:276] Timestamp: P: 1736672211711355 usec, L: 0
0112 08:57:03.906868 (+ 459us) write_op.cc:270] Start()
0112 08:57:03.906974 (+ 106us) write_op.cc:276] Timestamp: P: 1736672211735683 usec, L: 0
0112 08:57:03.907469 (+ 495us) write_op.cc:270] Start()
0112 08:57:03.907726 (+ 257us) write_op.cc:276] Timestamp: P: 1736672211812146 usec, L: 0
0112 08:57:03.908215 (+ 489us) write_op.cc:270] Start()
0112 08:57:03.908307 (+ 92us) write_op.cc:276] Timestamp: P: 1736672211842488 usec, L: 0
0112 08:57:03.908793 (+ 486us) write_op.cc:270] Start()
0112 08:57:03.908879 (+ 86us) write_op.cc:276] Timestamp: P: 1736672211895092 usec, L: 0
0112 08:57:03.909330 (+ 451us) write_op.cc:270] Start()
0112 08:57:03.909411 (+ 81us) write_op.cc:276] Timestamp: P: 1736672212045665 usec, L: 0
0112 08:57:03.909887 (+ 476us) write_op.cc:270] Start()
0112 08:57:03.909966 (+ 79us) write_op.cc:276] Timestamp: P: 1736672212078576 usec, L: 0
0112 08:57:03.910396 (+ 430us) write_op.cc:270] Start()
0112 08:57:03.910476 (+ 80us) write_op.cc:276] Timestamp: P: 1736672212085555 usec, L: 0
0112 08:57:03.910932 (+ 456us) write_op.cc:270] Start()
0112 08:57:03.911025 (+ 93us) write_op.cc:276] Timestamp: P: 1736672212143074 usec, L: 0
0112 08:57:03.911506 (+ 481us) write_op.cc:270] Start()
0112 08:57:03.911637 (+ 131us) write_op.cc:276] Timestamp: P: 1736672212154034 usec, L: 0
0112 08:57:03.919131 (+ 7494us) write_op.cc:270] Start()
0112 08:57:03.919226 (+ 95us) write_op.cc:276] Timestamp: P: 1736672212156216 usec, L: 0
0112 08:57:03.920348 (+ 1122us) write_op.cc:270] Start()
0112 08:57:03.920437 (+ 89us) write_op.cc:276] Timestamp: P: 1736672212158327 usec, L: 0
0112 08:57:03.920931 (+ 494us) write_op.cc:270] Start()
0112 08:57:03.921009 (+ 78us) write_op.cc:276] Timestamp: P: 1736672212218479 usec, L: 0
0112 08:57:03.921456 (+ 447us) write_op.cc:270] Start()
0112 08:57:03.921574 (+ 118us) write_op.cc:276] Timestamp: P: 1736672212254605 usec, L: 0
0112 08:57:03.928867 (+ 7293us) write_op.cc:270] Start()
0112 08:57:03.928962 (+ 95us) write_op.cc:276] Timestamp: P: 1736672212337246 usec, L: 0
0112 08:57:03.929484 (+ 522us) write_op.cc:270] Start()
0112 08:57:03.929572 (+ 88us) write_op.cc:276] Timestamp: P: 1736672212360577 usec, L: 0
0112 08:57:03.933614 (+ 4042us) write_op.cc:270] Start()
0112 08:57:03.933706 (+ 92us) write_op.cc:276] Timestamp: P: 1736672212363329 usec, L: 0
0112 08:57:03.934185 (+ 479us) write_op.cc:270] Start()
0112 08:57:03.934264 (+ 79us) write_op.cc:276] Timestamp: P: 1736672212367184 usec, L: 0
0112 08:57:03.934718 (+ 454us) write_op.cc:270] Start()
0112 08:57:03.934802 (+ 84us) write_op.cc:276] Timestamp: P: 1736672212469879 usec, L: 0
0112 08:57:03.935280 (+ 478us) write_op.cc:270] Start()
0112 08:57:03.935362 (+ 82us) write_op.cc:276] Timestamp: P: 1736672212476396 usec, L: 0
0112 08:57:03.935868 (+ 506us) write_op.cc:270] Start()
0112 08:57:03.935973 (+ 105us) write_op.cc:276] Timestamp: P: 1736672212481544 usec, L: 0
0112 08:57:03.936445 (+ 472us) write_op.cc:270] Start()
0112 08:57:03.936539 (+ 94us) write_op.cc:276] Timestamp: P: 1736672212517875 usec, L: 0
0112 08:57:03.937072 (+ 533us) write_op.cc:270] Start()
0112 08:57:03.937161 (+ 89us) write_op.cc:276] Timestamp: P: 1736672212523270 usec, L: 0
0112 08:57:03.937662 (+ 501us) write_op.cc:270] Start()
0112 08:57:03.937752 (+ 90us) write_op.cc:276] Timestamp: P: 1736672212640942 usec, L: 0
0112 08:57:03.938240 (+ 488us) write_op.cc:270] Start()
0112 08:57:03.938323 (+ 83us) write_op.cc:276] Timestamp: P: 1736672212646778 usec, L: 0
0112 08:57:03.938825 (+ 502us) write_op.cc:270] Start()
0112 08:57:04.193447 (+254622us) write_op.cc:276] Timestamp: P: 1736672212690363 usec, L: 0
0112 08:57:04.194104 (+ 657us) write_op.cc:270] Start()
0112 08:57:04.194198 (+ 94us) write_op.cc:276] Timestamp: P: 1736672212771498 usec, L: 0
0112 08:57:04.194820 (+ 622us) write_op.cc:270] Start()
0112 08:57:04.194909 (+ 89us) write_op.cc:276] Timestamp: P: 1736672213098801 usec, L: 0
0112 08:57:04.195399 (+ 490us) write_op.cc:270] Start()
0112 08:57:04.195487 (+ 88us) write_op.cc:276] Timestamp: P: 1736672213118435 usec, L: 0
0112 08:57:04.306930 (+111443us) write_op.cc:270] Start()
0112 08:57:04.307034 (+ 104us) write_op.cc:276] Timestamp: P: 1736672213121901 usec, L: 0
0112 08:57:04.307701 (+ 667us) write_op.cc:270] Start()
0112 08:57:04.307786 (+ 85us) write_op.cc:276] Timestamp: P: 1736672213125449 usec, L: 0
0112 08:57:04.310831 (+ 3045us) write_op.cc:270] Start()
0112 08:57:04.310911 (+ 80us) write_op.cc:276] Timestamp: P: 1736672213134275 usec, L: 0
0112 08:57:04.311427 (+ 516us) write_op.cc:270] Start()
0112 08:57:04.311501 (+ 74us) write_op.cc:276] Timestamp: P: 1736672213147190 usec, L: 0
0112 08:57:04.311960 (+ 459us) write_op.cc:270] Start()
0112 08:57:04.312031 (+ 71us) write_op.cc:276] Timestamp: P: 1736672213208633 usec, L: 0
0112 08:57:04.312469 (+ 438us) write_op.cc:270] Start()
0112 08:57:04.312539 (+ 70us) write_op.cc:276] Timestamp: P: 1736672213217253 usec, L: 0
0112 08:57:04.312960 (+ 421us) write_op.cc:270] Start()
0112 08:57:04.313042 (+ 82us) write_op.cc:276] Timestamp: P: 1736672213299632 usec, L: 0
0112 08:57:04.313492 (+ 450us) write_op.cc:270] Start()
0112 08:57:04.313561 (+ 69us) write_op.cc:276] Timestamp: P: 1736672213340031 usec, L: 0
0112 08:57:04.431270 (+117709us) write_op.cc:270] Start()
0112 08:57:04.431470 (+ 200us) write_op.cc:276] Timestamp: P: 1736672213386992 usec, L: 0
0112 08:57:04.433289 (+ 1819us) write_op.cc:270] Start()
0112 08:57:04.433395 (+ 106us) write_op.cc:276] Timestamp: P: 1736672213465650 usec, L: 0
0112 08:57:04.433928 (+ 533us) write_op.cc:270] Start()
0112 08:57:04.434016 (+ 88us) write_op.cc:276] Timestamp: P: 1736672213550318 usec, L: 0
0112 08:57:04.434506 (+ 490us) write_op.cc:270] Start()
0112 08:57:04.434587 (+ 81us) write_op.cc:276] Timestamp: P: 1736672213552394 usec, L: 0
0112 08:57:04.435088 (+ 501us) write_op.cc:270] Start()
0112 08:57:04.435183 (+ 95us) write_op.cc:276] Timestamp: P: 1736672213554351 usec, L: 0
0112 08:57:04.435668 (+ 485us) write_op.cc:270] Start()
0112 08:57:04.435747 (+ 79us) write_op.cc:276] Timestamp: P: 1736672213574861 usec, L: 0
0112 08:57:04.505066 (+ 69319us) write_op.cc:270] Start()
0112 08:57:04.505178 (+ 112us) write_op.cc:276] Timestamp: P: 1736672213593028 usec, L: 0
0112 08:57:04.505717 (+ 539us) write_op.cc:270] Start()
0112 08:57:04.505806 (+ 89us) write_op.cc:276] Timestamp: P: 1736672213632400 usec, L: 0
0112 08:57:04.506284 (+ 478us) write_op.cc:270] Start()
0112 08:57:04.506376 (+ 92us) write_op.cc:276] Timestamp: P: 1736672213638785 usec, L: 0
0112 08:57:04.507033 (+ 657us) write_op.cc:270] Start()
0112 08:57:04.507213 (+ 180us) write_op.cc:276] Timestamp: P: 1736672213646442 usec, L: 0
0112 08:57:04.507712 (+ 499us) write_op.cc:270] Start()
0112 08:57:04.507797 (+ 85us) write_op.cc:276] Timestamp: P: 1736672213727289 usec, L: 0
0112 08:57:04.508371 (+ 574us) write_op.cc:270] Start()
0112 08:57:04.508460 (+ 89us) write_op.cc:276] Timestamp: P: 1736672213872401 usec, L: 0
0112 08:57:04.508939 (+ 479us) write_op.cc:270] Start()
0112 08:57:04.509109 (+ 170us) write_op.cc:276] Timestamp: P: 1736672213917648 usec, L: 0
0112 08:57:04.514029 (+ 4920us) write_op.cc:270] Start()
0112 08:57:04.605896 (+ 91867us) write_op.cc:276] Timestamp: P: 1736672213920383 usec, L: 0
0112 08:57:04.606544 (+ 648us) write_op.cc:270] Start()
0112 08:57:04.606647 (+ 103us) write_op.cc:276] Timestamp: P: 1736672213933007 usec, L: 0
0112 08:57:04.607402 (+ 755us) write_op.cc:270] Start()
0112 08:57:04.607488 (+ 86us) write_op.cc:276] Timestamp: P: 1736672213943443 usec, L: 0
0112 08:57:04.608033 (+ 545us) write_op.cc:270] Start()
0112 08:57:04.608116 (+ 83us) write_op.cc:276] Timestamp: P: 1736672213971523 usec, L: 0
0112 08:57:04.608884 (+ 768us) write_op.cc:270] Start()
0112 08:57:04.608977 (+ 93us) write_op.cc:276] Timestamp: P: 1736672214004086 usec, L: 0
0112 08:57:04.609486 (+ 509us) write_op.cc:270] Start()
0112 08:57:04.609575 (+ 89us) write_op.cc:276] Timestamp: P: 1736672214006509 usec, L: 0
0112 08:57:04.610043 (+ 468us) write_op.cc:270] Start()
0112 08:57:04.610119 (+ 76us) write_op.cc:276] Timestamp: P: 1736672214043756 usec, L: 0
0112 08:57:04.610632 (+ 513us) write_op.cc:270] Start()
0112 08:57:04.610716 (+ 84us) write_op.cc:276] Timestamp: P: 1736672214244073 usec, L: 0
0112 08:57:04.611177 (+ 461us) write_op.cc:270] Start()
0112 08:57:04.611260 (+ 83us) write_op.cc:276] Timestamp: P: 1736672214402377 usec, L: 0
0112 08:57:04.611734 (+ 474us) write_op.cc:270] Start()
0112 08:57:04.611814 (+ 80us) write_op.cc:276] Timestamp: P: 1736672214405507 usec, L: 0
0112 08:57:04.644497 (+ 32683us) write_op.cc:270] Start()
0112 08:57:04.644596 (+ 99us) write_op.cc:276] Timestamp: P: 1736672214411130 usec, L: 0
0112 08:57:04.645616 (+ 1020us) write_op.cc:270] Start()
0112 08:57:04.645702 (+ 86us) write_op.cc:276] Timestamp: P: 1736672214417847 usec, L: 0
0112 08:57:04.646196 (+ 494us) write_op.cc:270] Start()
0112 08:57:04.646283 (+ 87us) write_op.cc:276] Timestamp: P: 1736672214447528 usec, L: 0
0112 08:57:04.646761 (+ 478us) write_op.cc:270] Start()
0112 08:57:04.646843 (+ 82us) write_op.cc:276] Timestamp: P: 1736672214450368 usec, L: 0
0112 08:57:04.647310 (+ 467us) write_op.cc:270] Start()
0112 08:57:04.647399 (+ 89us) write_op.cc:276] Timestamp: P: 1736672214464469 usec, L: 0
0112 08:57:04.656444 (+ 9045us) write_op.cc:270] Start()
0112 08:57:04.656528 (+ 84us) write_op.cc:276] Timestamp: P: 1736672214568255 usec, L: 0
0112 08:57:04.656997 (+ 469us) write_op.cc:270] Start()
0112 08:57:04.657078 (+ 81us) write_op.cc:276] Timestamp: P: 1736672214570763 usec, L: 0
0112 08:57:04.657536 (+ 458us) write_op.cc:270] Start()
0112 08:57:04.657627 (+ 91us) write_op.cc:276] Timestamp: P: 1736672214645967 usec, L: 0
0112 08:57:04.658115 (+ 488us) write_op.cc:270] Start()
0112 08:57:04.670743 (+ 12628us) write_op.cc:276] Timestamp: P: 1736672214678890 usec, L: 0
0112 08:57:04.675044 (+ 4301us) write_op.cc:270] Start()
0112 08:57:04.675142 (+ 98us) write_op.cc:276] Timestamp: P: 1736672214720018 usec, L: 0
0112 08:57:04.675699 (+ 557us) write_op.cc:270] Start()
0112 08:57:04.675787 (+ 88us) write_op.cc:276] Timestamp: P: 1736672214751441 usec, L: 0
0112 08:57:04.681493 (+ 5706us) write_op.cc:270] Start()
0112 08:57:04.681582 (+ 89us) write_op.cc:276] Timestamp: P: 1736672214757003 usec, L: 0
0112 08:57:04.682057 (+ 475us) write_op.cc:270] Start()
0112 08:57:04.682140 (+ 83us) write_op.cc:276] Timestamp: P: 1736672214779924 usec, L: 0
0112 08:57:04.682623 (+ 483us) write_op.cc:270] Start()
0112 08:57:04.682705 (+ 82us) write_op.cc:276] Timestamp: P: 1736672214823561 usec, L: 0
0112 08:57:04.683157 (+ 452us) write_op.cc:270] Start()
0112 08:57:04.683233 (+ 76us) write_op.cc:276] Timestamp: P: 1736672214843507 usec, L: 0
0112 08:57:04.683740 (+ 507us) write_op.cc:270] Start()
0112 08:57:04.683825 (+ 85us) write_op.cc:276] Timestamp: P: 1736672214845475 usec, L: 0
0112 08:57:04.708346 (+ 24521us) write_op.cc:270] Start()
0112 08:57:04.708459 (+ 113us) write_op.cc:276] Timestamp: P: 1736672214933374 usec, L: 0
0112 08:57:04.709014 (+ 555us) write_op.cc:270] Start()
0112 08:57:04.709095 (+ 81us) write_op.cc:276] Timestamp: P: 1736672214949734 usec, L: 0
0112 08:57:04.709569 (+ 474us) write_op.cc:270] Start()
0112 08:57:04.709648 (+ 79us) write_op.cc:276] Timestamp: P: 1736672214956908 usec, L: 0
0112 08:57:04.710104 (+ 456us) write_op.cc:270] Start()
0112 08:57:04.710178 (+ 74us) write_op.cc:276] Timestamp: P: 1736672214988608 usec, L: 0
0112 08:57:04.710630 (+ 452us) write_op.cc:270] Start()
0112 08:57:04.710713 (+ 83us) write_op.cc:276] Timestamp: P: 1736672215033698 usec, L: 0
0112 08:57:04.711175 (+ 462us) write_op.cc:270] Start()
0112 08:57:04.711256 (+ 81us) write_op.cc:276] Timestamp: P: 1736672215126098 usec, L: 0
0112 08:57:04.711723 (+ 467us) write_op.cc:270] Start()
0112 08:57:04.711805 (+ 82us) write_op.cc:276] Timestamp: P: 1736672215158602 usec, L: 0
0112 08:57:04.723429 (+ 11624us) write_op.cc:270] Start()
0112 08:57:04.723521 (+ 92us) write_op.cc:276] Timestamp: P: 1736672215180810 usec, L: 0
0112 08:57:04.725392 (+ 1871us) write_op.cc:270] Start()
0112 08:57:04.725494 (+ 102us) write_op.cc:276] Timestamp: P: 1736672215238347 usec, L: 0
0112 08:57:04.726127 (+ 633us) write_op.cc:270] Start()
0112 08:57:04.726212 (+ 85us) write_op.cc:276] Timestamp: P: 1736672215320937 usec, L: 0
0112 08:57:04.726709 (+ 497us) write_op.cc:270] Start()
0112 08:57:04.726803 (+ 94us) write_op.cc:276] Timestamp: P: 1736672215339093 usec, L: 0
0112 08:57:04.727288 (+ 485us) write_op.cc:270] Start()
0112 08:57:04.727518 (+ 230us) write_op.cc:276] Timestamp: P: 1736672215431574 usec, L: 0
0112 08:57:04.728158 (+ 640us) write_op.cc:270] Start()
0112 08:57:04.728247 (+ 89us) write_op.cc:276] Timestamp: P: 1736672215482447 usec, L: 0
0112 08:57:04.728736 (+ 489us) write_op.cc:270] Start()
0112 08:57:04.728820 (+ 84us) write_op.cc:276] Timestamp: P: 1736672215580729 usec, L: 0
0112 08:57:04.729298 (+ 478us) write_op.cc:270] Start()
0112 08:57:04.729391 (+ 93us) write_op.cc:276] Timestamp: P: 1736672215586556 usec, L: 0
0112 08:57:04.729843 (+ 452us) write_op.cc:270] Start()
0112 08:57:04.729926 (+ 83us) write_op.cc:276] Timestamp: P: 1736672215615790 usec, L: 0
0112 08:57:04.730377 (+ 451us) write_op.cc:270] Start()
0112 08:57:04.730450 (+ 73us) write_op.cc:276] Timestamp: P: 1736672215747405 usec, L: 0
0112 08:57:04.748507 (+ 18057us) write_op.cc:270] Start()
0112 08:57:04.748617 (+ 110us) write_op.cc:276] Timestamp: P: 1736672215889721 usec, L: 0
0112 08:57:04.749166 (+ 549us) write_op.cc:270] Start()
0112 08:57:04.749252 (+ 86us) write_op.cc:276] Timestamp: P: 1736672215957762 usec, L: 0
0112 08:57:04.749749 (+ 497us) write_op.cc:270] Start()
0112 08:57:04.749836 (+ 87us) write_op.cc:276] Timestamp: P: 1736672216027132 usec, L: 0
0112 08:57:04.750493 (+ 657us) write_op.cc:270] Start()
0112 08:57:04.750582 (+ 89us) write_op.cc:276] Timestamp: P: 1736672216112859 usec, L: 0
0112 08:57:04.751101 (+ 519us) write_op.cc:270] Start()
0112 08:57:04.751190 (+ 89us) write_op.cc:276] Timestamp: P: 1736672216247036 usec, L: 0
0112 08:57:04.751672 (+ 482us) write_op.cc:270] Start()
0112 08:57:04.751752 (+ 80us) write_op.cc:276] Timestamp: P: 1736672216384427 usec, L: 0
0112 08:57:04.764251 (+ 12499us) write_op.cc:270] Start()
0112 08:57:04.764363 (+ 112us) write_op.cc:276] Timestamp: P: 1736672216464160 usec, L: 0
0112 08:57:04.764908 (+ 545us) write_op.cc:270] Start()
0112 08:57:04.764994 (+ 86us) write_op.cc:276] Timestamp: P: 1736672216582262 usec, L: 0
0112 08:57:04.765557 (+ 563us) write_op.cc:270] Start()
0112 08:57:04.765646 (+ 89us) write_op.cc:276] Timestamp: P: 1736672216652602 usec, L: 0
0112 08:57:04.766127 (+ 481us) write_op.cc:270] Start()
0112 08:57:04.766208 (+ 81us) write_op.cc:276] Timestamp: P: 1736672216654277 usec, L: 0
0112 08:57:04.766707 (+ 499us) write_op.cc:270] Start()
0112 08:57:04.766791 (+ 84us) write_op.cc:276] Timestamp: P: 1736672216705977 usec, L: 0
0112 08:57:04.767247 (+ 456us) write_op.cc:270] Start()
0112 08:57:04.767327 (+ 80us) write_op.cc:276] Timestamp: P: 1736672216707853 usec, L: 0
0112 08:57:04.767994 (+ 667us) write_op.cc:270] Start()
0112 08:57:04.768087 (+ 93us) write_op.cc:276] Timestamp: P: 1736672216717628 usec, L: 0
0112 08:57:04.768561 (+ 474us) write_op.cc:270] Start()
0112 08:57:04.768642 (+ 81us) write_op.cc:276] Timestamp: P: 1736672216719584 usec, L: 0
0112 08:57:04.769131 (+ 489us) write_op.cc:270] Start()
0112 08:57:04.769212 (+ 81us) write_op.cc:276] Timestamp: P: 1736672216725440 usec, L: 0
0112 08:57:04.769671 (+ 459us) write_op.cc:270] Start()
0112 08:57:04.769751 (+ 80us) write_op.cc:276] Timestamp: P: 1736672216910593 usec, L: 0
0112 08:57:04.770207 (+ 456us) write_op.cc:270] Start()
0112 08:57:04.770280 (+ 73us) write_op.cc:276] Timestamp: P: 1736672216929049 usec, L: 0
0112 08:57:04.770735 (+ 455us) write_op.cc:270] Start()
0112 08:57:04.770817 (+ 82us) write_op.cc:276] Timestamp: P: 1736672216933440 usec, L: 0
0112 08:57:04.790711 (+ 19894us) write_op.cc:270] Start()
0112 08:57:04.790837 (+ 126us) write_op.cc:276] Timestamp: P: 1736672216949078 usec, L: 0
0112 08:57:04.791481 (+ 644us) write_op.cc:270] Start()
0112 08:57:04.791583 (+ 102us) write_op.cc:276] Timestamp: P: 1736672216953235 usec, L: 0
0112 08:57:04.792094 (+ 511us) write_op.cc:270] Start()
0112 08:57:04.792175 (+ 81us) write_op.cc:276] Timestamp: P: 1736672216956809 usec, L: 0
0112 08:57:04.792643 (+ 468us) write_op.cc:270] Start()
0112 08:57:04.792723 (+ 80us) write_op.cc:276] Timestamp: P: 1736672216979311 usec, L: 0
0112 08:57:04.793162 (+ 439us) write_op.cc:270] Start()
0112 08:57:04.793239 (+ 77us) write_op.cc:276] Timestamp: P: 1736672216991322 usec, L: 0
0112 08:57:04.793744 (+ 505us) write_op.cc:270] Start()
0112 08:57:04.793824 (+ 80us) write_op.cc:276] Timestamp: P: 1736672217011121 usec, L: 0
0112 08:57:04.794276 (+ 452us) write_op.cc:270] Start()
0112 08:57:04.794358 (+ 82us) write_op.cc:276] Timestamp: P: 1736672217021632 usec, L: 0
0112 08:57:04.794971 (+ 613us) write_op.cc:270] Start()
0112 08:57:04.795051 (+ 80us) write_op.cc:276] Timestamp: P: 1736672217282050 usec, L: 0
0112 08:57:04.795525 (+ 474us) write_op.cc:270] Start()
0112 08:57:04.795593 (+ 68us) write_op.cc:276] Timestamp: P: 1736672217316835 usec, L: 0
0112 08:57:04.796015 (+ 422us) write_op.cc:270] Start()
0112 08:57:04.796185 (+ 170us) write_op.cc:276] Timestamp: P: 1736672217374956 usec, L: 0
0112 08:57:04.796661 (+ 476us) write_op.cc:270] Start()
0112 08:57:04.796745 (+ 84us) write_op.cc:276] Timestamp: P: 1736672217376696 usec, L: 0
0112 08:57:04.797209 (+ 464us) write_op.cc:270] Start()
0112 08:57:04.797293 (+ 84us) write_op.cc:276] Timestamp: P: 1736672217412633 usec, L: 0
0112 08:57:04.797777 (+ 484us) write_op.cc:270] Start()
0112 08:57:04.814696 (+ 16919us) write_op.cc:276] Timestamp: P: 1736672217486656 usec, L: 0
0112 08:57:04.816971 (+ 2275us) write_op.cc:270] Start()
0112 08:57:04.817077 (+ 106us) write_op.cc:276] Timestamp: P: 1736672217503730 usec, L: 0
0112 08:57:04.822485 (+ 5408us) write_op.cc:270] Start()
0112 08:57:04.822578 (+ 93us) write_op.cc:276] Timestamp: P: 1736672217509934 usec, L: 0
0112 08:57:04.825846 (+ 3268us) write_op.cc:270] Start()
0112 08:57:04.825936 (+ 90us) write_op.cc:276] Timestamp: P: 1736672217550298 usec, L: 0
0112 08:57:04.826447 (+ 511us) write_op.cc:270] Start()
0112 08:57:04.826530 (+ 83us) write_op.cc:276] Timestamp: P: 1736672217644129 usec, L: 0
0112 08:57:04.826988 (+ 458us) write_op.cc:270] Start()
0112 08:57:04.827078 (+ 90us) write_op.cc:276] Timestamp: P: 1736672217696170 usec, L: 0
0112 08:57:04.827585 (+ 507us) write_op.cc:270] Start()
0112 08:57:04.827668 (+ 83us) write_op.cc:276] Timestamp: P: 1736672217744024 usec, L: 0
0112 08:57:04.828166 (+ 498us) write_op.cc:270] Start()
0112 08:57:04.828249 (+ 83us) write_op.cc:276] Timestamp: P: 1736672217778163 usec, L: 0
0112 08:57:04.828746 (+ 497us) write_op.cc:270] Start()
0112 08:57:04.828820 (+ 74us) write_op.cc:276] Timestamp: P: 1736672217810036 usec, L: 0
0112 08:57:04.829228 (+ 408us) write_op.cc:270] Start()
0112 08:57:04.829299 (+ 71us) write_op.cc:276] Timestamp: P: 1736672217850465 usec, L: 0
0112 08:57:04.844172 (+ 14873us) write_op.cc:270] Start()
0112 08:57:04.844274 (+ 102us) write_op.cc:276] Timestamp: P: 1736672217861259 usec, L: 0
0112 08:57:04.844827 (+ 553us) write_op.cc:270] Start()
0112 08:57:04.844922 (+ 95us) write_op.cc:276] Timestamp: P: 1736672217913262 usec, L: 0
0112 08:57:04.845426 (+ 504us) write_op.cc:270] Start()
0112 08:57:04.845507 (+ 81us) write_op.cc:276] Timestamp: P: 1736672218011047 usec, L: 0
0112 08:57:04.845984 (+ 477us) write_op.cc:270] Start()
0112 08:57:04.846068 (+ 84us) write_op.cc:276] Timestamp: P: 1736672218013766 usec, L: 0
0112 08:57:04.846557 (+ 489us) write_op.cc:270] Start()
0112 08:57:04.846639 (+ 82us) write_op.cc:276] Timestamp: P: 1736672218026050 usec, L: 0
0112 08:57:04.847111 (+ 472us) write_op.cc:270] Start()
0112 08:57:04.847186 (+ 75us) write_op.cc:276] Timestamp: P: 1736672218177174 usec, L: 0
0112 08:57:04.847653 (+ 467us) write_op.cc:270] Start()
0112 08:57:04.847727 (+ 74us) write_op.cc:276] Timestamp: P: 1736672218315210 usec, L: 0
0112 08:57:04.859018 (+ 11291us) write_op.cc:270] Start()
0112 08:57:04.859112 (+ 94us) write_op.cc:276] Timestamp: P: 1736672218434528 usec, L: 0
0112 08:57:04.859650 (+ 538us) write_op.cc:270] Start()
0112 08:57:04.859730 (+ 80us) write_op.cc:276] Timestamp: P: 1736672218498462 usec, L: 0
0112 08:57:04.860238 (+ 508us) write_op.cc:270] Start()
0112 08:57:04.860315 (+ 77us) write_op.cc:276] Timestamp: P: 1736672218715942 usec, L: 0
0112 08:57:04.860817 (+ 502us) write_op.cc:270] Start()
0112 08:57:04.860899 (+ 82us) write_op.cc:276] Timestamp: P: 1736672218719563 usec, L: 0
0112 08:57:04.861313 (+ 414us) write_op.cc:270] Start()
0112 08:57:04.861396 (+ 83us) write_op.cc:276] Timestamp: P: 1736672218736978 usec, L: 0
0112 08:57:04.861832 (+ 436us) write_op.cc:270] Start()
0112 08:57:04.861900 (+ 68us) write_op.cc:276] Timestamp: P: 1736672218739858 usec, L: 0
0112 08:57:04.862298 (+ 398us) write_op.cc:270] Start()
0112 08:57:04.862378 (+ 80us) write_op.cc:276] Timestamp: P: 1736672218890267 usec, L: 0
0112 08:57:04.862804 (+ 426us) write_op.cc:270] Start()
0112 08:57:04.862874 (+ 70us) write_op.cc:276] Timestamp: P: 1736672218914605 usec, L: 0
0112 08:57:04.863308 (+ 434us) write_op.cc:270] Start()
0112 08:57:04.863401 (+ 93us) write_op.cc:276] Timestamp: P: 1736672219060656 usec, L: 0
0112 08:57:04.898039 (+ 34638us) write_op.cc:270] Start()
0112 08:57:04.898133 (+ 94us) write_op.cc:276] Timestamp: P: 1736672219114412 usec, L: 0
0112 08:57:04.898634 (+ 501us) write_op.cc:270] Start()
0112 08:57:04.898717 (+ 83us) write_op.cc:276] Timestamp: P: 1736672219231398 usec, L: 0
0112 08:57:04.899158 (+ 441us) write_op.cc:270] Start()
0112 08:57:04.912856 (+ 13698us) write_op.cc:276] Timestamp: P: 1736672219485915 usec, L: 0
0112 08:57:04.913493 (+ 637us) write_op.cc:270] Start()
0112 08:57:04.913586 (+ 93us) write_op.cc:276] Timestamp: P: 1736672219614934 usec, L: 0
0112 08:57:04.914065 (+ 479us) write_op.cc:270] Start()
0112 08:57:04.914131 (+ 66us) write_op.cc:276] Timestamp: P: 1736672219694589 usec, L: 0
0112 08:57:04.914561 (+ 430us) write_op.cc:270] Start()
0112 08:57:04.914621 (+ 60us) write_op.cc:276] Timestamp: P: 1736672219740754 usec, L: 0
0112 08:57:04.915027 (+ 406us) write_op.cc:270] Start()
0112 08:57:04.915090 (+ 63us) write_op.cc:276] Timestamp: P: 1736672219775573 usec, L: 0
0112 08:57:04.915571 (+ 481us) write_op.cc:270] Start()
0112 08:57:04.915644 (+ 73us) write_op.cc:276] Timestamp: P: 1736672219880669 usec, L: 0
0112 08:57:04.916089 (+ 445us) write_op.cc:270] Start()
0112 08:57:04.916159 (+ 70us) write_op.cc:276] Timestamp: P: 1736672220048492 usec, L: 0
0112 08:57:04.916628 (+ 469us) write_op.cc:270] Start()
0112 08:57:04.916692 (+ 64us) write_op.cc:276] Timestamp: P: 1736672220053456 usec, L: 0
0112 08:57:04.917073 (+ 381us) write_op.cc:270] Start()
0112 08:57:04.917141 (+ 68us) write_op.cc:276] Timestamp: P: 1736672220089314 usec, L: 0
0112 08:57:04.917540 (+ 399us) write_op.cc:270] Start()
0112 08:57:04.917610 (+ 70us) write_op.cc:276] Timestamp: P: 1736672220181678 usec, L: 0
0112 08:57:04.918025 (+ 415us) write_op.cc:270] Start()
0112 08:57:04.918096 (+ 71us) write_op.cc:276] Timestamp: P: 1736672220202205 usec, L: 0
0112 08:57:04.918484 (+ 388us) write_op.cc:270] Start()
0112 08:57:04.918556 (+ 72us) write_op.cc:276] Timestamp: P: 1736672220364269 usec, L: 0
0112 08:57:04.918952 (+ 396us) write_op.cc:270] Start()
0112 08:57:04.919018 (+ 66us) write_op.cc:276] Timestamp: P: 1736672220378324 usec, L: 0
0112 08:57:04.919425 (+ 407us) write_op.cc:270] Start()
0112 08:57:04.919489 (+ 64us) write_op.cc:276] Timestamp: P: 1736672220475135 usec, L: 0
0112 08:57:04.968718 (+ 49229us) write_op.cc:270] Start()
0112 08:57:04.968835 (+ 117us) write_op.cc:276] Timestamp: P: 1736672220477077 usec, L: 0
0112 08:57:04.969555 (+ 720us) write_op.cc:270] Start()
0112 08:57:04.969648 (+ 93us) write_op.cc:276] Timestamp: P: 1736672220481746 usec, L: 0
0112 08:57:04.970091 (+ 443us) write_op.cc:270] Start()
0112 08:57:04.970183 (+ 92us) write_op.cc:276] Timestamp: P: 1736672220505456 usec, L: 0
0112 08:57:04.970632 (+ 449us) write_op.cc:270] Start()
0112 08:57:04.970699 (+ 67us) write_op.cc:276] Timestamp: P: 1736672220538091 usec, L: 0
0112 08:57:04.971109 (+ 410us) write_op.cc:270] Start()
0112 08:57:04.971184 (+ 75us) write_op.cc:276] Timestamp: P: 1736672220649806 usec, L: 0
0112 08:57:04.971653 (+ 469us) write_op.cc:270] Start()
0112 08:57:04.971746 (+ 93us) write_op.cc:276] Timestamp: P: 1736672220722251 usec, L: 0
0112 08:57:04.972180 (+ 434us) write_op.cc:270] Start()
0112 08:57:04.972261 (+ 81us) write_op.cc:276] Timestamp: P: 1736672220733209 usec, L: 0
0112 08:57:04.972667 (+ 406us) write_op.cc:270] Start()
0112 08:57:04.972735 (+ 68us) write_op.cc:276] Timestamp: P: 1736672220843289 usec, L: 0
0112 08:57:04.973229 (+ 494us) write_op.cc:270] Start()
0112 08:57:04.973311 (+ 82us) write_op.cc:276] Timestamp: P: 1736672220846674 usec, L: 0
0112 08:57:04.973769 (+ 458us) write_op.cc:270] Start()
0112 08:57:04.973836 (+ 67us) write_op.cc:276] Timestamp: P: 1736672220999249 usec, L: 0
0112 08:57:04.974277 (+ 441us) write_op.cc:270] Start()
0112 08:57:04.974350 (+ 73us) write_op.cc:276] Timestamp: P: 1736672221127570 usec, L: 0
0112 08:57:04.974773 (+ 423us) write_op.cc:270] Start()
0112 08:57:04.974846 (+ 73us) write_op.cc:276] Timestamp: P: 1736672221129460 usec, L: 0
0112 08:57:04.975268 (+ 422us) write_op.cc:270] Start()
0112 08:57:04.975341 (+ 73us) write_op.cc:276] Timestamp: P: 1736672221180278 usec, L: 0
0112 08:57:04.975786 (+ 445us) write_op.cc:270] Start()
0112 08:57:04.975858 (+ 72us) write_op.cc:276] Timestamp: P: 1736672221308729 usec, L: 0
0112 08:57:05.010418 (+ 34560us) write_op.cc:270] Start()
0112 08:57:05.010517 (+ 99us) write_op.cc:276] Timestamp: P: 1736672221345055 usec, L: 0
0112 08:57:05.010968 (+ 451us) write_op.cc:270] Start()
0112 08:57:05.011031 (+ 63us) write_op.cc:276] Timestamp: P: 1736672221435187 usec, L: 0
0112 08:57:05.011441 (+ 410us) write_op.cc:270] Start()
0112 08:57:05.011512 (+ 71us) write_op.cc:276] Timestamp: P: 1736672221492835 usec, L: 0
0112 08:57:05.011857 (+ 345us) write_op.cc:270] Start()
0112 08:57:05.011940 (+ 83us) write_op.cc:276] Timestamp: P: 1736672221524703 usec, L: 0
0112 08:57:05.012408 (+ 468us) write_op.cc:270] Start()
0112 08:57:05.012494 (+ 86us) write_op.cc:276] Timestamp: P: 1736672221531702 usec, L: 0
0112 08:57:05.012962 (+ 468us) write_op.cc:270] Start()
0112 08:57:05.013026 (+ 64us) write_op.cc:276] Timestamp: P: 1736672221538352 usec, L: 0
0112 08:57:05.013396 (+ 370us) write_op.cc:270] Start()
0112 08:57:05.013456 (+ 60us) write_op.cc:276] Timestamp: P: 1736672221602157 usec, L: 0
0112 08:57:05.013820 (+ 364us) write_op.cc:270] Start()
0112 08:57:05.013896 (+ 76us) write_op.cc:276] Timestamp: P: 1736672
I20250112 08:57:09.346338 21675 master-stress-test.cc:577] Killing test thread 1/10
I20250112 08:57:09.361711 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 200) took 1074 ms. Trace:
I20250112 08:57:09.362107 22557 rpcz_store.cc:276] 0112 08:57:08.287418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.305682 (+ 18264us) service_pool.cc:225] Handling call
0112 08:57:09.361391 (+1055709us) ts_tablet_manager.cc:1145] Deleting tablet ea15d9405b9944db8e00121965b9ffde
0112 08:57:09.361436 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.361630 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054886}
W20250112 08:57:09.219058 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet fc82026fcd1541418176a015ea3253cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc82026fcd1541418176a015ea3253cf already in progress: creating tablet
W20250112 08:57:09.369827 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 150810us
I20250112 08:57:09.220441 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a5959caa4d39420981e3b1994df4cb17 (table default.Table_275569887e7c49dd9608ddf9efc7fb39 [id=8172f1c5f3b54191aaa2322ccd8c638d]). Expected version 1 got 0
W20250112 08:57:09.370781 21717 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3352b4abff244e88e533d442b24437d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:09.370951 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:09.371335 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet a72d2062bf2a4c79b5a8be974d1b8cb5 (table default.Table_eb6046dddf9e4a28a8254e74a7cce16a [id=78a10e1e86624d089f2b6866a4a2da03]). Expected version 2 got 1
W20250112 08:57:09.370502 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet f9835288a7974e85821b095c03e91886 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9835288a7974e85821b095c03e91886 already in progress: creating tablet
I20250112 08:57:09.372671 21709 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:09.374578 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 92a22f87fbbf48e88dbd5af418ee94ad (table default.Table_ec36ed57a80f4350bda6894141741ba3 [id=6cb07113f32f4f039b2f200179a14796]) successfully deleted
W20250112 08:57:09.378952 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet ea15d9405b9944db8e00121965b9ffde because the tablet was not found. No further retry: Not found: Tablet not found: ea15d9405b9944db8e00121965b9ffde
I20250112 08:57:09.367569 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet 267ab3e5955b4d3bbeca44baf021b328 with delete_type TABLET_DATA_DELETED (replaced by tablet 340fcbf7e3cf4038b5bed1b188da18dd) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:09.383407 23099 raft_consensus.cc:491] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.384409 23099 raft_consensus.cc:513] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.386194 23099 leader_election.cc:290] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.389225 22456 raft_consensus.cc:357] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.390046 22456 raft_consensus.cc:383] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.392503 22557 tablet_replica.cc:331] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:09.408075 22461 tablet_bootstrap.cc:654] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.415894 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b5d9d5141941b69b33cc3f5d8b9ec0" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.422004 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6368a80a9fe9457a803e68a2392b638d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.424378 22356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d246b7d414aadbb69ac692a5d40c7" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.427647 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c34fc4b6aaf45fdae76cf153129f2da" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.429873 22350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e924f523524bd0a387597399e8470a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.430052 22351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.430739 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62bae16ab43414c8d8be2766802d4c7" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.431504 22349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd228f7a432b436581368da7eab3ef16" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:09.442903 23161 tablet.cc:1721] T c9848b879a2e40aeb840924261e6a100 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:09.445894 22565 log.cc:1198] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:09.446674 22565 ts_tablet_manager.cc:1939] T 06717cb567a54ee6a268c3be5d8786ec P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:09.447402 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 144) took 6356 ms. Trace:
I20250112 08:57:09.447691 22565 rpcz_store.cc:276] 0112 08:57:03.090671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:03.094751 (+ 4080us) service_pool.cc:225] Handling call
0112 08:57:09.080594 (+5985843us) ts_tablet_manager.cc:1145] Deleting tablet 06717cb567a54ee6a268c3be5d8786ec
0112 08:57:09.087191 (+ 6597us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.179980 (+ 92789us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.263373 (+ 83393us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.447342 (+183969us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11375,"tablet-delete.queue_time_us":5985364}
I20250112 08:57:09.265954 22458 consensus_queue.cc:260] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.267448 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd228f7a432b436581368da7eab3ef16" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:09.449085 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 06717cb567a54ee6a268c3be5d8786ec (table default.Table_c18e0bb841d942978709c4d445929483 [id=4e7fa7e2c76f4932bd3ae514033670e1]) successfully deleted
I20250112 08:57:09.299043 23131 raft_consensus.cc:491] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c608785d6874abf9cc28db11a979ddd)
I20250112 08:57:09.449777 23131 raft_consensus.cc:513] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:09.452512 22146 leader_election.cc:343] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.431982 22062 raft_consensus.cc:3054] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:09.440254 23169 tablet.cc:1721] T c9848b879a2e40aeb840924261e6a100 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:09.454535 23131 leader_election.cc:290] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.459936 22002 leader_election.cc:304] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.460886 23131 raft_consensus.cc:3054] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:09.461670 22002 leader_election.cc:304] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.475436 22458 ts_tablet_manager.cc:1428] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.271s user 0.002s sys 0.005s
W20250112 08:57:09.479264 21999 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:09.483964 22458 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:09.487263 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea15d9405b9944db8e00121965b9ffde" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:09.491307 22068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a22f87fbbf48e88dbd5af418ee94ad" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:09.492803 22063 raft_consensus.cc:3054] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:09.502418 22002 leader_election.cc:343] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:09.502874 22002 leader_election.cc:304] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.501794 23099 raft_consensus.cc:491] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.504593 23193 tablet.cc:1721] T a72d2062bf2a4c79b5a8be974d1b8cb5 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:09.505146 23099 raft_consensus.cc:513] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.508430 23099 leader_election.cc:290] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:09.526060 22290 consensus_peers.cc:487] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:09.526760 22290 consensus_peers.cc:487] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:09.527307 22290 leader_election.cc:343] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.531020 23136 raft_consensus.cc:2743] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:09.557531 23186 raft_consensus.cc:491] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53068c99960647728141706e9c50b870)
I20250112 08:57:09.558208 23186 raft_consensus.cc:513] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:09.390332 22456 raft_consensus.cc:738] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.576637 22456 consensus_queue.cc:260] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:09.604067 22290 leader_election.cc:343] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:09.604779 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 76880.5us
I20250112 08:57:09.568131 23096 raft_consensus.cc:2743] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:09.561069 23186 leader_election.cc:290] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:09.617694 23203 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:09.621096 23203 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
W20250112 08:57:09.605504 21999 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 5157642d667d49aaad41df56be38801f
I20250112 08:57:09.606585 23131 raft_consensus.cc:2743] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:09.629734 22290 consensus_peers.cc:487] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:09.633690 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15d38686b2c64b20ba87275e3ba176ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } } }
I20250112 08:57:09.634583 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:09.644234 22565 tablet_replica.cc:331] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:09.645025 22565 raft_consensus.cc:2238] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:09.645931 22565 raft_consensus.cc:2267] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:09.650357 23204 raft_consensus.cc:491] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.650781 23204 raft_consensus.cc:513] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.652880 22456 ts_tablet_manager.cc:1428] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.426s user 0.007s sys 0.000s
W20250112 08:57:09.666770 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 177620us
I20250112 08:57:09.595007 22458 tablet_bootstrap.cc:654] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.681771 21675 master-stress-test.cc:579] Killed test thread 1/10
I20250112 08:57:09.682034 21675 master-stress-test.cc:577] Killing test thread 2/10
I20250112 08:57:09.682273 21675 master-stress-test.cc:579] Killed test thread 2/10
I20250112 08:57:09.682472 21675 master-stress-test.cc:577] Killing test thread 3/10
I20250112 08:57:09.682693 21675 master-stress-test.cc:579] Killed test thread 3/10
I20250112 08:57:09.682894 21675 master-stress-test.cc:577] Killing test thread 4/10
I20250112 08:57:09.683145 21675 master-stress-test.cc:579] Killed test thread 4/10
I20250112 08:57:09.683346 21675 master-stress-test.cc:577] Killing test thread 5/10
I20250112 08:57:09.690660 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:09.692435 21717 master_service.cc:946] ReplaceTablet: received request to replace tablet c49ec037c22a42108ecb48bc8270191f from {username='slave'} at 127.0.0.1:56650
I20250112 08:57:09.697598 21802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:09.697839 23203 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:09.673131 22456 tablet_bootstrap.cc:492] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:09.712631 23204 leader_election.cc:290] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.714960 23162 raft_consensus.cc:491] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.717051 23162 raft_consensus.cc:513] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:09.717464 23197 log.cc:926] Time spent T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20250112 08:57:09.718746 23162 leader_election.cc:290] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.724128 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 65591a0bb44148c49715b4564dd13524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:56 UTC) from {username='slave'} at 127.0.0.1:56484
W20250112 08:57:09.725461 21709 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 43) took 1079 ms (client timeout 1000 ms). Trace:
W20250112 08:57:09.725831 21709 rpcz_store.cc:269] 0112 08:57:08.646285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.659288 (+ 13003us) service_pool.cc:225] Handling call
0112 08:57:09.725382 (+1066094us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":13}
I20250112 08:57:09.647776 22565 ts_tablet_manager.cc:1905] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:09.706933 23099 consensus_queue.cc:1035] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250112 08:57:09.739738 23099 raft_consensus.cc:491] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:57:09.739737 22002 leader_election.cc:343] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: 5157642d667d49aaad41df56be38801f
I20250112 08:57:09.740233 22002 leader_election.cc:304] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.740171 23099 raft_consensus.cc:513] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.742691 23099 leader_election.cc:290] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.715291 23188 consensus_queue.cc:1035] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250112 08:57:09.730548 23204 raft_consensus.cc:491] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.746853 23204 raft_consensus.cc:513] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.748580 23204 leader_election.cc:290] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.755985 23099 raft_consensus.cc:491] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.756496 23099 raft_consensus.cc:513] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.758371 23099 leader_election.cc:290] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:09.762959 23103 ts_tablet_manager.cc:1918] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:09.763435 23103 log.cc:1198] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/5cd402674080464cb3bde3697ac23180
I20250112 08:57:09.764181 23103 ts_tablet_manager.cc:1939] T 5cd402674080464cb3bde3697ac23180 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:09.766147 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 194) took 2057 ms. Trace:
I20250112 08:57:09.766749 22456 tablet_bootstrap.cc:654] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.767401 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:57:09.773048 23203 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:09.810348 22557 raft_consensus.cc:2238] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:09.810837 22557 raft_consensus.cc:2267] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:09.816542 22557 ts_tablet_manager.cc:1905] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:09.834560 22456 tablet_bootstrap.cc:492] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:09.834990 22456 ts_tablet_manager.cc:1397] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.162s user 0.007s sys 0.003s
I20250112 08:57:09.841557 22557 ts_tablet_manager.cc:1918] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:09.841918 22557 log.cc:1198] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/2e62435dce1441a3b2fee50518662d48
I20250112 08:57:09.782393 22461 tablet_bootstrap.cc:492] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:09.842504 22557 ts_tablet_manager.cc:1939] T 2e62435dce1441a3b2fee50518662d48 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:09.842803 22461 ts_tablet_manager.cc:1397] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.522s user 0.018s sys 0.006s
I20250112 08:57:09.842578 23103 rpcz_store.cc:276] 0112 08:57:07.709016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.710359 (+ 1343us) service_pool.cc:225] Handling call
0112 08:57:08.275222 (+564863us) ts_tablet_manager.cc:1145] Deleting tablet 5cd402674080464cb3bde3697ac23180
0112 08:57:08.476279 (+201057us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.004601 (+528322us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.762749 (+758148us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.766089 (+ 3340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":564361}
I20250112 08:57:09.844182 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 198) took 1982 ms. Trace:
I20250112 08:57:09.844594 23103 rpcz_store.cc:276] 0112 08:57:07.861722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.861855 (+ 133us) service_pool.cc:225] Handling call
0112 08:57:09.843801 (+1981946us) ts_tablet_manager.cc:1145] Deleting tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9
0112 08:57:09.843843 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.844127 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981386}
I20250112 08:57:09.845978 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 200) took 1809 ms. Trace:
I20250112 08:57:09.845803 22461 raft_consensus.cc:357] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.846473 22461 raft_consensus.cc:383] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.846747 22461 raft_consensus.cc:738] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.846390 23103 rpcz_store.cc:276] 0112 08:57:08.036818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.038622 (+ 1804us) service_pool.cc:225] Handling call
0112 08:57:09.845666 (+1807044us) ts_tablet_manager.cc:1145] Deleting tablet 010ce6f55c444fe0852ad72cc7383cc1
0112 08:57:09.845722 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.845918 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806482}
I20250112 08:57:09.847361 22461 consensus_queue.cc:260] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.850386 22461 ts_tablet_manager.cc:1428] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250112 08:57:09.852775 22461 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:09.854992 22456 raft_consensus.cc:357] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:09.855695 22456 raft_consensus.cc:383] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.855948 22456 raft_consensus.cc:738] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:57:09.831146 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 200232us
W20250112 08:57:09.860033 22290 consensus_peers.cc:487] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 422e1484cd84479abba824cc8a6fae1f. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:09.860684 22290 leader_election.cc:343] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 92a22f87fbbf48e88dbd5af418ee94ad
I20250112 08:57:09.862910 22290 leader_election.cc:304] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:09.863713 23188 raft_consensus.cc:2743] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:09.864168 22456 consensus_queue.cc:260] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:57:09.864683 22290 leader_election.cc:343] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: ea15d9405b9944db8e00121965b9ffde
I20250112 08:57:09.868587 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 202) took 1580 ms. Trace:
I20250112 08:57:09.869016 22290 leader_election.cc:304] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:09.868921 22557 rpcz_store.cc:276] 0112 08:57:08.288435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.306686 (+ 18251us) service_pool.cc:225] Handling call
0112 08:57:09.392272 (+1085586us) ts_tablet_manager.cc:1145] Deleting tablet 2e62435dce1441a3b2fee50518662d48
0112 08:57:09.392331 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.825553 (+433222us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.841360 (+ 15807us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.868527 (+ 27167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":417048,"tablet-delete.queue_time_us":1084810}
I20250112 08:57:09.870189 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 2e62435dce1441a3b2fee50518662d48 (table default.Table_835c7ebe5c7d4b3a92a8c7124cbd48b6 [id=ce7c89cc8d89473cada34608b03819a5]) successfully deleted
I20250112 08:57:09.870647 23188 raft_consensus.cc:2743] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:09.870182 22461 tablet_bootstrap.cc:654] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.877571 23193 tablet.cc:1721] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 08:57:09.879375 22149 leader_election.cc:343] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.879729 22149 leader_election.cc:304] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.880829 23206 raft_consensus.cc:2743] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:09.881754 22149 leader_election.cc:343] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.882079 22149 leader_election.cc:304] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.882944 23206 raft_consensus.cc:2743] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:09.883941 22149 leader_election.cc:343] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.884310 22149 leader_election.cc:304] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:09.885401 23206 raft_consensus.cc:2743] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:09.847748 23103 tablet_replica.cc:331] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:09.887400 22039 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:09.872570 23131 raft_consensus.cc:2743] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:09.899434 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da16d10565a74243a9ab8c5264bf7b1a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:09.900843 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b34b09f56d4417a3f321599170e5fd" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:09.902731 22146 leader_election.cc:343] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:09.903360 22146 leader_election.cc:343] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.906199 23206 raft_consensus.cc:491] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:09.906635 23206 raft_consensus.cc:513] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:09.908478 23206 leader_election.cc:290] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:09.910388 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9835288a7974e85821b095c03e91886" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:09.915462 22146 leader_election.cc:343] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:09.924093 22456 ts_tablet_manager.cc:1428] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.089s user 0.004s sys 0.004s
I20250112 08:57:09.925491 22565 ts_tablet_manager.cc:1918] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:09.925843 22565 log.cc:1198] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/253b5b6ea6f044fc8dcaafdc2e587f84
I20250112 08:57:09.926540 22565 ts_tablet_manager.cc:1939] T 253b5b6ea6f044fc8dcaafdc2e587f84 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:09.927453 22458 tablet_bootstrap.cc:492] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:09.929430 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 201) took 1641 ms. Trace:
I20250112 08:57:09.930018 22458 ts_tablet_manager.cc:1397] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.446s user 0.024s sys 0.002s
I20250112 08:57:09.929772 22557 rpcz_store.cc:276] 0112 08:57:08.287725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.309845 (+ 22120us) service_pool.cc:225] Handling call
0112 08:57:09.929127 (+1619282us) ts_tablet_manager.cc:1145] Deleting tablet ac1889a1ab4f4d8aa4f7915d91f2cf27
0112 08:57:09.929179 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.929369 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618512}
W20250112 08:57:09.932271 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 because the tablet was not found. No further retry: Not found: Tablet not found: ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:09.932577 22458 raft_consensus.cc:357] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:09.933182 22458 raft_consensus.cc:383] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:09.933431 22458 raft_consensus.cc:738] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:09.938421 22456 tablet_bootstrap.cc:492] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:09.944924 22458 consensus_queue.cc:260] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:09.965890 22557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37418 (request call id 208) took 1505 ms. Trace:
I20250112 08:57:09.966347 22557 rpcz_store.cc:276] 0112 08:57:08.460822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.460956 (+ 134us) service_pool.cc:225] Handling call
0112 08:57:09.952158 (+1491202us) ts_tablet_manager.cc:1145] Deleting tablet 1ad1f85596894fe9992ec4ad115e6928
0112 08:57:09.952198 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.965826 (+ 13628us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490644}
W20250112 08:57:09.967695 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 1ad1f85596894fe9992ec4ad115e6928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad1f85596894fe9992ec4ad115e6928 already in progress: creating tablet
I20250112 08:57:09.975626 22458 ts_tablet_manager.cc:1428] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.045s user 0.004s sys 0.000s
I20250112 08:57:09.928498 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 157) took 5316 ms. Trace:
I20250112 08:57:09.976871 22565 rpcz_store.cc:276] 0112 08:57:04.611707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.611819 (+ 112us) service_pool.cc:225] Handling call
0112 08:57:09.644077 (+5032258us) ts_tablet_manager.cc:1145] Deleting tablet 253b5b6ea6f044fc8dcaafdc2e587f84
0112 08:57:09.644130 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:09.912198 (+268068us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.925281 (+ 13083us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:09.928438 (+ 3157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5031636}
I20250112 08:57:09.978015 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 253b5b6ea6f044fc8dcaafdc2e587f84 (table default.Table_de19595082ed4dc890bbceb724ea6b19 [id=46cec39657e342aea69c4a97774466bd]) successfully deleted
I20250112 08:57:09.989938 22565 tablet_replica.cc:331] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:09.991016 22456 tablet_bootstrap.cc:654] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:09.991379 22458 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:09.995988 22565 raft_consensus.cc:2238] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:09.996677 22565 raft_consensus.cc:2267] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:09.998340 22565 ts_tablet_manager.cc:1905] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:10.007649 22458 tablet_bootstrap.cc:654] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:10.010244 22149 leader_election.cc:343] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.010643 22149 leader_election.cc:304] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:10.011584 23206 raft_consensus.cc:2743] T b62bae16ab43414c8d8be2766802d4c7 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.020519 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1889a1ab4f4d8aa4f7915d91f2cf27" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:10.023021 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e62435dce1441a3b2fee50518662d48" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:10.024479 22149 leader_election.cc:343] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.024758 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:10.024873 22149 leader_election.cc:304] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:10.025372 22209 raft_consensus.cc:2463] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
W20250112 08:57:10.025799 22149 leader_election.cc:343] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.026157 22149 leader_election.cc:304] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:10.028594 22182 tablet_service.cc:1467] Processing CreateTablet for tablet 13990acd16404d119298d020b6d14a4e (DEFAULT_TABLE table=default.Table_ba481b6297ae48a4b20921031aef6276 [id=a991cdf15fe34001a629f0a82c393d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.029685 22182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13990acd16404d119298d020b6d14a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.023530 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:10.035604 22192 tablet_service.cc:1467] Processing CreateTablet for tablet e65ae6f4959b4a57a7357e1040f8dec8 (DEFAULT_TABLE table=default.Table_fcb376f8af89430c84a3f2c0fea366b1 [id=1ab96aa394664871bf9cf220d2390d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.036761 22192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65ae6f4959b4a57a7357e1040f8dec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.051622 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 13990acd16404d119298d020b6d14a4e (DEFAULT_TABLE table=default.Table_ba481b6297ae48a4b20921031aef6276 [id=a991cdf15fe34001a629f0a82c393d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.051822 22039 tablet_service.cc:1467] Processing CreateTablet for tablet e65ae6f4959b4a57a7357e1040f8dec8 (DEFAULT_TABLE table=default.Table_fcb376f8af89430c84a3f2c0fea366b1 [id=1ab96aa394664871bf9cf220d2390d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.052742 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13990acd16404d119298d020b6d14a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.056582 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65ae6f4959b4a57a7357e1040f8dec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.066869 21708 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet 575891fa458d4d8391d493dbe34fc0b3
I20250112 08:57:10.069675 21708 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56650 (request call id 295) took 1378 ms. Trace:
I20250112 08:57:10.069700 22183 tablet_service.cc:1514] Processing DeleteTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:10.072671 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 575891fa458d4d8391d493dbe34fc0b3 because the tablet was not found. No further retry: Not found: Tablet not found: 575891fa458d4d8391d493dbe34fc0b3
I20250112 08:57:10.074422 22043 tablet_service.cc:1514] Processing DeleteTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:10.075456 22044 tablet_service.cc:1467] Processing CreateTablet for tablet ffa0f45a8ee94ad69b403509cb532ed3 (DEFAULT_TABLE table=default.Table_6280d02d39bc47eba2925dcc1fef5835 [id=4c44ec3226a44bef89713c95fed68652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.076442 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa0f45a8ee94ad69b403509cb532ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.081490 22183 tablet_service.cc:1467] Processing CreateTablet for tablet ffa0f45a8ee94ad69b403509cb532ed3 (DEFAULT_TABLE table=default.Table_6280d02d39bc47eba2925dcc1fef5835 [id=4c44ec3226a44bef89713c95fed68652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.082871 22183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa0f45a8ee94ad69b403509cb532ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.084794 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 5a162200945646729e1744cd6fd407c3 (DEFAULT_TABLE table=default.Table_4198997d702f4c4bb04a3082456851c8 [id=f6593a616557410586bf02e03726f0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.089740 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a162200945646729e1744cd6fd407c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.096170 23165 raft_consensus.cc:2743] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.097249 23165 raft_consensus.cc:2743] T 6368a80a9fe9457a803e68a2392b638d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:10.098835 22149 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.195:33805, user_credentials={real_user=slave}} blocked reactor thread for 73053.8us
I20250112 08:57:10.102669 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 8d28a29fb18843c79df8f58743c6b0b0 (DEFAULT_TABLE table=default.Table_2055fd4bf7224371ba88109fd2e0f629 [id=c0f115b088be4be8b5a6cbba28785a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.103734 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d28a29fb18843c79df8f58743c6b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.109709 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 5a162200945646729e1744cd6fd407c3 (DEFAULT_TABLE table=default.Table_4198997d702f4c4bb04a3082456851c8 [id=f6593a616557410586bf02e03726f0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.109836 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 8d28a29fb18843c79df8f58743c6b0b0 (DEFAULT_TABLE table=default.Table_2055fd4bf7224371ba88109fd2e0f629 [id=c0f115b088be4be8b5a6cbba28785a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.110684 22040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d28a29fb18843c79df8f58743c6b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.111204 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a162200945646729e1744cd6fd407c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:10.122856 21704 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 267ab3e5955b4d3bbeca44baf021b328 consensus state has no leader
I20250112 08:57:10.131847 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18dfd51dbe40f9830c9f873347375d (DEFAULT_TABLE table=default.Table_33acf77ade0f4b5d8e348dfe6e38b4fa [id=f6ce4a5bdd694fc7bdda81f7e020e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.133384 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18dfd51dbe40f9830c9f873347375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.136152 22458 tablet_bootstrap.cc:492] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:10.136595 22458 ts_tablet_manager.cc:1397] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.160s user 0.012s sys 0.000s
I20250112 08:57:10.136845 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 221796ee0715492eaf4c50619e55e2fc (DEFAULT_TABLE table=default.Table_9b44fe9ca083478fa41484d20bf2a49d [id=e021f84a29554f89a70beb984932c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.137804 22191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221796ee0715492eaf4c50619e55e2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.138928 22458 raft_consensus.cc:357] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.139564 22458 raft_consensus.cc:383] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.150089 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 221796ee0715492eaf4c50619e55e2fc (DEFAULT_TABLE table=default.Table_9b44fe9ca083478fa41484d20bf2a49d [id=e021f84a29554f89a70beb984932c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.151229 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221796ee0715492eaf4c50619e55e2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.154381 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa44c174012470d80fd00970bb27883 (DEFAULT_TABLE table=default.Table_b639de28dc9642f29a1f0caeecc05a57 [id=d15d4c0cf8e940a7a857afa2ae304749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.154628 22189 tablet_service.cc:1467] Processing CreateTablet for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8 (DEFAULT_TABLE table=default.Table_6872481b576147d695568e3d5bf35e77 [id=321f8586dfc440ff9262f12bbf0e882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.155396 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa44c174012470d80fd00970bb27883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.155959 22189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.161248 22048 tablet_service.cc:1467] Processing CreateTablet for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8 (DEFAULT_TABLE table=default.Table_6872481b576147d695568e3d5bf35e77 [id=321f8586dfc440ff9262f12bbf0e882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.162165 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.168995 22036 tablet_service.cc:1467] Processing CreateTablet for tablet 38baf231bceb4591b3d3e86006542f3a (DEFAULT_TABLE table=default.Table_44de87cd557a4c878db8d37210254226 [id=2a955059b25e440d861c5635c041a8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.170140 22036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38baf231bceb4591b3d3e86006542f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.174770 22043 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18dfd51dbe40f9830c9f873347375d (DEFAULT_TABLE table=default.Table_33acf77ade0f4b5d8e348dfe6e38b4fa [id=f6ce4a5bdd694fc7bdda81f7e020e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.175830 22043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18dfd51dbe40f9830c9f873347375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.139851 22458 raft_consensus.cc:738] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.180603 22458 consensus_queue.cc:260] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.182929 22458 ts_tablet_manager.cc:1428] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20250112 08:57:10.183815 22458 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:10.185822 22565 ts_tablet_manager.cc:1918] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:10.193231 22565 log.cc:1198] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:10.193960 22565 ts_tablet_manager.cc:1939] T 0b38deeff1964b09938a12cfd4992ec5 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:10.186745 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa44c174012470d80fd00970bb27883 (DEFAULT_TABLE table=default.Table_b639de28dc9642f29a1f0caeecc05a57 [id=d15d4c0cf8e940a7a857afa2ae304749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.199441 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet a5959caa4d39420981e3b1994df4cb17 (table default.Table_275569887e7c49dd9608ddf9efc7fb39 [id=8172f1c5f3b54191aaa2322ccd8c638d]). Expected version 1 got 0
I20250112 08:57:10.200170 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:10.200909 21715 catalog_manager.cc:5526] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:10.204602 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa44c174012470d80fd00970bb27883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.211151 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 38baf231bceb4591b3d3e86006542f3a (DEFAULT_TABLE table=default.Table_44de87cd557a4c878db8d37210254226 [id=2a955059b25e440d861c5635c041a8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.212154 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38baf231bceb4591b3d3e86006542f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.212049 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 1722022567174f7a8cc9c85fe6655b4c (DEFAULT_TABLE table=default.Table_6ba47fafc72c4861b7adf19ac9fba3cf [id=c6661e6ba3d64dc7af7c461c79b4934f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.213541 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1722022567174f7a8cc9c85fe6655b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.213804 22187 tablet_service.cc:1467] Processing CreateTablet for tablet cde9fbb824d040ca90b4109703790810 (DEFAULT_TABLE table=default.Table_00c8e2048c444eeca4149e2b643a7abe [id=a5a69fa5f6df4391ac5c0ee7288668cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.214797 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde9fbb824d040ca90b4109703790810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.216082 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 158) took 5603 ms. Trace:
I20250112 08:57:10.219993 22565 rpcz_store.cc:276] 0112 08:57:04.612843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:04.612939 (+ 96us) service_pool.cc:225] Handling call
0112 08:57:09.989771 (+5376832us) ts_tablet_manager.cc:1145] Deleting tablet 0b38deeff1964b09938a12cfd4992ec5
0112 08:57:09.989825 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.049454 (+ 59629us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.185637 (+136183us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.216018 (+ 30381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5376329}
I20250112 08:57:10.221642 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 137e2275af51421cac4ca629dfa809ba (DEFAULT_TABLE table=default.Table_60d957dcf38841e5a1ab61b409df19af [id=3390616d770745a39ee27f66c84ce788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.222744 22185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e2275af51421cac4ca629dfa809ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.222932 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 167) took 5125 ms. Trace:
I20250112 08:57:10.223237 22565 rpcz_store.cc:276] 0112 08:57:05.143971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.144230 (+ 259us) service_pool.cc:225] Handling call
0112 08:57:10.222604 (+5078374us) ts_tablet_manager.cc:1145] Deleting tablet 5c3e34e1d70a4c63956a5be660e02b21
0112 08:57:10.222647 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.222882 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077507}
I20250112 08:57:10.226194 22458 tablet_bootstrap.cc:654] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.240389 22045 tablet_service.cc:1467] Processing CreateTablet for tablet ae66efbd303d4a25b4c547915109bca7 (DEFAULT_TABLE table=default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.239840 22038 tablet_service.cc:1467] Processing CreateTablet for tablet cde9fbb824d040ca90b4109703790810 (DEFAULT_TABLE table=default.Table_00c8e2048c444eeca4149e2b643a7abe [id=a5a69fa5f6df4391ac5c0ee7288668cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.240231 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 137e2275af51421cac4ca629dfa809ba (DEFAULT_TABLE table=default.Table_60d957dcf38841e5a1ab61b409df19af [id=3390616d770745a39ee27f66c84ce788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.241499 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae66efbd303d4a25b4c547915109bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.242136 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde9fbb824d040ca90b4109703790810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.244418 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e2275af51421cac4ca629dfa809ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.252099 23204 raft_consensus.cc:491] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.252262 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 1722022567174f7a8cc9c85fe6655b4c (DEFAULT_TABLE table=default.Table_6ba47fafc72c4861b7adf19ac9fba3cf [id=c6661e6ba3d64dc7af7c461c79b4934f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.252540 23204 raft_consensus.cc:513] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.253278 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1722022567174f7a8cc9c85fe6655b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.254361 23204 leader_election.cc:290] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:10.255957 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:10.257500 22146 leader_election.cc:343] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.226881 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 169) took 5082 ms. Trace:
I20250112 08:57:10.277604 22565 rpcz_store.cc:276] 0112 08:57:05.144867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.146368 (+ 1501us) service_pool.cc:225] Handling call
0112 08:57:10.226558 (+5080190us) ts_tablet_manager.cc:1145] Deleting tablet f46707de727a49108afb12ea2cf08aa0
0112 08:57:10.226597 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.226822 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5079455}
I20250112 08:57:10.279731 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 171) took 5134 ms. Trace:
I20250112 08:57:10.269902 23210 raft_consensus.cc:491] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.297962 23210 raft_consensus.cc:513] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.297176 22565 rpcz_store.cc:276] 0112 08:57:05.145602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.147262 (+ 1660us) service_pool.cc:225] Handling call
0112 08:57:10.279363 (+5132101us) ts_tablet_manager.cc:1145] Deleting tablet 54b34b09f56d4417a3f321599170e5fd
0112 08:57:10.279412 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.279678 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5131377}
I20250112 08:57:10.303258 22565 tablet_replica.cc:331] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:10.303967 22565 raft_consensus.cc:2238] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:10.304440 22565 raft_consensus.cc:2267] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:10.320282 23209 raft_consensus.cc:491] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.321094 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c6ea87589249739461e629eee5f40b" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:10.320801 23209 raft_consensus.cc:513] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.325765 23209 leader_election.cc:290] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:10.329005 22349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b34b09f56d4417a3f321599170e5fd" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:10.329416 22354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da16d10565a74243a9ab8c5264bf7b1a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:10.330818 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9835288a7974e85821b095c03e91886" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:10.332530 22350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:10.333230 22350 raft_consensus.cc:2463] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:10.261977 21717 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet c49ec037c22a42108ecb48bc8270191f
I20250112 08:57:10.348531 23210 leader_election.cc:290] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:10.349363 21717 catalog_manager.cc:6596] ReplaceTablet: tablet c49ec037c22a42108ecb48bc8270191f deleted and replaced by tablet b7c3c7162ff144a79715944b7c9f0371
I20250112 08:57:10.070098 21708 rpcz_store.cc:276] 0112 08:57:08.691692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.991753 (+300061us) service_pool.cc:225] Handling call
0112 08:57:09.216327 (+224574us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:57:09.285752 (+ 69425us) catalog_manager.cc:2753] Modifying in-memory table state
0112 08:57:09.285900 (+ 148us) catalog_manager.cc:2762] Locking tablets
0112 08:57:10.000717 (+714817us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 08:57:10.065908 (+ 65191us) catalog_manager.cc:2794] Removing table from by-name map
0112 08:57:10.066670 (+ 762us) catalog_manager.cc:2810] Committing in-memory state
0112 08:57:10.066785 (+ 115us) catalog_manager.cc:2814] Aborting table tasks
0112 08:57:10.069595 (+ 2810us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:10.012237 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:10.012583 (+ 346us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:10.012607 (+ 24us) write_op.cc:435] Acquired schema lock
0112 08:57:10.012620 (+ 13us) tablet.cc:659] Decoding operations
0112 08:57:10.012936 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:10.012980 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:10.013010 (+ 30us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:57:10.013282 (+ 272us) tablet.cc:698] Row locks acquired
0112 08:57:10.013295 (+ 13us) write_op.cc:260] PREPARE: finished
0112 08:57:10.013408 (+ 113us) write_op.cc:270] Start()
0112 08:57:10.013498 (+ 90us) write_op.cc:276] Timestamp: P: 1736672230013345 usec, L: 0
0112 08:57:10.013516 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 08:57:10.014034 (+ 518us) log.cc:844] Serialized 731 byte log entry
0112 08:57:10.054456 (+ 40422us) op_driver.cc:464] REPLICATION: finished
0112 08:57:10.054807 (+ 351us) write_op.cc:301] APPLY: starting
0112 08:57:10.054857 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:10.054970 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:10.055056 (+ 86us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:10.055312 (+ 256us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:10.055461 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 08:57:10.055500 (+ 39us) write_op.cc:312] APPLY: finished
0112 08:57:10.055664 (+ 164us) log.cc:844] Serialized 29 byte log entry
0112 08:57:10.055806 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.055868 (+ 62us) write_op.cc:454] Released schema lock
0112 08:57:10.055900 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:10.056120 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.056156 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":209,"apply.run_cpu_time_us":1285,"apply.run_wall_time_us":1284,"num_ops":2,"prepare.queue_time_us":9568,"prepare.run_cpu_time_us":2074,"prepare.run_wall_time_us":2074,"raft.queue_time_us":209,"raft.run_cpu_time_us":733,"raft.run_wall_time_us":734,"replication_time_us":40841}]]}
I20250112 08:57:10.326678 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 5cd402674080464cb3bde3697ac23180 (table default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059]) successfully deleted
I20250112 08:57:10.351354 22565 ts_tablet_manager.cc:1905] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:10.351341 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 already in progress: creating tablet
W20250112 08:57:10.352074 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 010ce6f55c444fe0852ad72cc7383cc1 because the tablet was not found. No further retry: Not found: Tablet not found: 010ce6f55c444fe0852ad72cc7383cc1
I20250112 08:57:10.343479 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 0b38deeff1964b09938a12cfd4992ec5 (table default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]) successfully deleted
W20250112 08:57:10.365331 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 5c3e34e1d70a4c63956a5be660e02b21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c3e34e1d70a4c63956a5be660e02b21 already in progress: creating tablet
I20250112 08:57:10.368167 22461 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:10.368670 22461 ts_tablet_manager.cc:1397] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.516s user 0.013s sys 0.035s
I20250112 08:57:10.351332 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:10.380996 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:10.393114 23193 tablet.cc:1721] T c9848b879a2e40aeb840924261e6a100 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:10.396085 22458 tablet_bootstrap.cc:492] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:10.396523 22458 ts_tablet_manager.cc:1397] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.213s user 0.011s sys 0.006s
I20250112 08:57:10.398833 22458 raft_consensus.cc:357] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.399505 22458 raft_consensus.cc:383] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.399746 22458 raft_consensus.cc:738] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.400403 22458 consensus_queue.cc:260] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.401180 22565 ts_tablet_manager.cc:1918] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250112 08:57:10.401546 22565 log.cc:1198] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/5157642d667d49aaad41df56be38801f
W20250112 08:57:10.394076 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet f46707de727a49108afb12ea2cf08aa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f46707de727a49108afb12ea2cf08aa0 already in progress: creating tablet
I20250112 08:57:10.402217 22565 ts_tablet_manager.cc:1939] T 5157642d667d49aaad41df56be38801f P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:10.402031 22461 raft_consensus.cc:357] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:10.402514 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 54b34b09f56d4417a3f321599170e5fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54b34b09f56d4417a3f321599170e5fd already in progress: creating tablet
I20250112 08:57:10.402712 22461 raft_consensus.cc:383] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.402923 22458 ts_tablet_manager.cc:1428] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 08:57:10.402956 22461 raft_consensus.cc:738] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.403954 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:10.407126 22188 tablet_service.cc:1467] Processing CreateTablet for tablet ae66efbd303d4a25b4c547915109bca7 (DEFAULT_TABLE table=default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.408335 22188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae66efbd303d4a25b4c547915109bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.409495 21715 catalog_manager.cc:5526] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:10.414273 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a16774d8e4bee8ae6cbdc760d003a (DEFAULT_TABLE table=default.Table_79dac8e9ea5d483397bd1ec1cbc74403 [id=dad0bc3b967c4dec94f065c3d35df670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.415360 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5a16774d8e4bee8ae6cbdc760d003a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.429453 22181 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a16774d8e4bee8ae6cbdc760d003a (DEFAULT_TABLE table=default.Table_79dac8e9ea5d483397bd1ec1cbc74403 [id=dad0bc3b967c4dec94f065c3d35df670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.430490 22181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5a16774d8e4bee8ae6cbdc760d003a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.438017 22458 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:10.441929 23225 tablet.cc:1721] T 815d246b7d414aadbb69ac692a5d40c7 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:10.442552 22456 tablet_bootstrap.cc:492] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:10.442940 22458 tablet_bootstrap.cc:654] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.442929 22456 ts_tablet_manager.cc:1397] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.505s user 0.005s sys 0.006s
I20250112 08:57:10.446070 22456 raft_consensus.cc:357] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:10.446718 22456 raft_consensus.cc:383] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.446964 22456 raft_consensus.cc:738] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.447631 22456 consensus_queue.cc:260] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:10.451562 22456 ts_tablet_manager.cc:1428] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250112 08:57:10.454764 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 3b164f5284a4484db5b03af09ec68276 (DEFAULT_TABLE table=default.Table_660015f62b6c4db59a358f76526c5dad [id=f55e906f67104045afdabcb6b80c4345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.455802 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b164f5284a4484db5b03af09ec68276. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:10.456327 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 125801us
W20250112 08:57:10.457231 22290 leader_election.cc:343] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:10.457810 22290 leader_election.cc:343] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:10.458309 22290 leader_election.cc:343] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.459358 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 166) took 5400 ms. Trace:
I20250112 08:57:10.459666 22565 rpcz_store.cc:276] 0112 08:57:05.059377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.082374 (+ 22997us) service_pool.cc:225] Handling call
0112 08:57:10.303099 (+5220725us) ts_tablet_manager.cc:1145] Deleting tablet 5157642d667d49aaad41df56be38801f
0112 08:57:10.303146 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.370613 (+ 67467us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.400978 (+ 30365us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.459293 (+ 58315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5142396}
I20250112 08:57:10.460876 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 170) took 5315 ms. Trace:
I20250112 08:57:10.461174 22565 rpcz_store.cc:276] 0112 08:57:05.145260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.169491 (+ 24231us) service_pool.cc:225] Handling call
0112 08:57:10.460642 (+5291151us) ts_tablet_manager.cc:1145] Deleting tablet c08341b5ce6040d5be30f324d0373e38
0112 08:57:10.460676 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.460831 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5256152}
I20250112 08:57:10.462239 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 173) took 5208 ms. Trace:
I20250112 08:57:10.462523 22565 rpcz_store.cc:276] 0112 08:57:05.253349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.255119 (+ 1770us) service_pool.cc:225] Handling call
0112 08:57:10.461931 (+5206812us) ts_tablet_manager.cc:1145] Deleting tablet 80c6ea87589249739461e629eee5f40b
0112 08:57:10.461963 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.462196 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5206195}
I20250112 08:57:10.463430 22565 tablet_replica.cc:331] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:10.464619 22565 raft_consensus.cc:2238] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:10.465006 22565 raft_consensus.cc:2267] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:10.475777 22565 ts_tablet_manager.cc:1905] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:10.478672 22290 leader_election.cc:343] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.480528 23103 raft_consensus.cc:2238] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:10.481216 23103 raft_consensus.cc:2267] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:10.483489 23103 ts_tablet_manager.cc:1905] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:10.404610 22461 consensus_queue.cc:260] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.486225 22461 ts_tablet_manager.cc:1428] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.117s user 0.002s sys 0.004s
I20250112 08:57:10.487387 22461 tablet_bootstrap.cc:492] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:10.504309 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:10.506850 22149 leader_election.cc:343] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.507217 22149 leader_election.cc:304] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:10.508140 22149 leader_election.cc:343] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.508466 22149 leader_election.cc:304] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:10.509506 22149 leader_election.cc:343] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.509892 22149 leader_election.cc:304] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:10.511888 22067 raft_consensus.cc:3054] T f61aedea18d944179e788c08488c0a5e P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:10.515435 23165 raft_consensus.cc:2743] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.514664 21715 catalog_manager.cc:5526] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:10.516048 23165 raft_consensus.cc:2743] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.516929 23165 raft_consensus.cc:2743] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.518489 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet c49ec037c22a42108ecb48bc8270191f with delete_type TABLET_DATA_DELETED (replaced by tablet b7c3c7162ff144a79715944b7c9f0371) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:10.518680 22048 tablet_service.cc:1514] Processing DeleteTablet for tablet c49ec037c22a42108ecb48bc8270191f with delete_type TABLET_DATA_DELETED (replaced by tablet b7c3c7162ff144a79715944b7c9f0371) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:10.520428 22036 tablet_service.cc:1467] Processing CreateTablet for tablet 340fcbf7e3cf4038b5bed1b188da18dd (DEFAULT_TABLE table=default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.522023 22036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340fcbf7e3cf4038b5bed1b188da18dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.523777 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:10.522961 22149 leader_election.cc:304] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:10.528018 23196 raft_consensus.cc:2798] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:10.528357 23196 raft_consensus.cc:491] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:10.528650 23196 raft_consensus.cc:3054] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:10.532547 23204 raft_consensus.cc:491] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.532938 23204 raft_consensus.cc:513] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:10.534785 23204 leader_election.cc:290] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:10.536185 22063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e80e259db4645942141754c8491a3" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:10.537724 22146 leader_election.cc:343] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.538702 23131 raft_consensus.cc:491] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.539120 23131 raft_consensus.cc:513] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:10.540967 23131 leader_election.cc:290] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:10.542610 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 3b164f5284a4484db5b03af09ec68276 (DEFAULT_TABLE table=default.Table_660015f62b6c4db59a358f76526c5dad [id=f55e906f67104045afdabcb6b80c4345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.544734 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44cad3bd5a84e75a41733d8d2f345a8" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:10.545557 22355 raft_consensus.cc:2463] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:10.546723 22002 leader_election.cc:304] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:10.547482 23131 raft_consensus.cc:2798] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:10.547372 23232 tablet.cc:1721] T 815d246b7d414aadbb69ac692a5d40c7 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:10.547822 23131 raft_consensus.cc:491] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:10.548161 23131 raft_consensus.cc:3054] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:10.554515 23131 raft_consensus.cc:513] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
W20250112 08:57:10.560014 22290 leader_election.cc:343] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:10.560135 23196 raft_consensus.cc:513] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.562523 22458 tablet_bootstrap.cc:492] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:10.560384 22290 leader_election.cc:304] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:10.564774 23209 raft_consensus.cc:2743] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.565867 22290 leader_election.cc:304] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:10.564725 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:10.566114 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 340fcbf7e3cf4038b5bed1b188da18dd (DEFAULT_TABLE table=default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:10.567095 22146 leader_election.cc:343] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.567711 23196 leader_election.cc:290] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:10.566175 22458 ts_tablet_manager.cc:1397] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.007s
I20250112 08:57:10.570717 23209 raft_consensus.cc:2798] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:10.571072 23209 raft_consensus.cc:491] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:10.571372 23209 raft_consensus.cc:3054] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:10.574819 22458 raft_consensus.cc:357] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.575884 22458 raft_consensus.cc:383] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.576216 22458 raft_consensus.cc:738] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.578969 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 (DEFAULT_TABLE table=default.Table_55a599c370f441969aad1da2ebfb4793 [id=14d632c40d984bd3aa1930f930705313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.578151 23209 raft_consensus.cc:513] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.578339 22045 tablet_service.cc:1467] Processing CreateTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 (DEFAULT_TABLE table=default.Table_55a599c370f441969aad1da2ebfb4793 [id=14d632c40d984bd3aa1930f930705313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.582957 23209 leader_election.cc:290] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:10.579705 22048 tablet_service.cc:1467] Processing CreateTablet for tablet 8509c81d4fd54b5dbca4633433f2eb48 (DEFAULT_TABLE table=default.Table_1871a9c14e194ccfae3a16776b48ee2e [id=186484f3ebf14b3193d2baec31cdb49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.584105 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8509c81d4fd54b5dbca4633433f2eb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.582484 22458 consensus_queue.cc:260] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.587970 23216 tablet.cc:1721] T a5959caa4d39420981e3b1994df4cb17 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 08:57:10.592989 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575891fa458d4d8391d493dbe34fc0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.601665 22458 ts_tablet_manager.cc:1428] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.033s user 0.002s sys 0.003s
I20250112 08:57:10.613550 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 5157642d667d49aaad41df56be38801f (table default.Table_a0ed97a52d8140f1869abb6a3095add6 [id=e812d5ec0441481bb47d2aa1eeee407b]) successfully deleted
W20250112 08:57:10.614732 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet c08341b5ce6040d5be30f324d0373e38 because the tablet was not found. No further retry: Not found: Tablet not found: c08341b5ce6040d5be30f324d0373e38
I20250112 08:57:10.615218 22458 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:10.615869 23103 ts_tablet_manager.cc:1918] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:10.616451 23103 log.cc:1198] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/422e1484cd84479abba824cc8a6fae1f
I20250112 08:57:10.617588 23103 ts_tablet_manager.cc:1939] T 422e1484cd84479abba824cc8a6fae1f P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:10.620587 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 201) took 2535 ms. Trace:
I20250112 08:57:10.620918 23103 rpcz_store.cc:276] 0112 08:57:08.085489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.085620 (+ 131us) service_pool.cc:225] Handling call
0112 08:57:09.847553 (+1761933us) ts_tablet_manager.cc:1145] Deleting tablet 422e1484cd84479abba824cc8a6fae1f
0112 08:57:09.847604 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.585557 (+737953us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.615538 (+ 29981us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.620522 (+ 4984us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":209293,"tablet-delete.queue_time_us":1761383}
I20250112 08:57:10.622864 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 204) took 2348 ms. Trace:
I20250112 08:57:10.623306 23103 rpcz_store.cc:276] 0112 08:57:08.274164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.274268 (+ 104us) service_pool.cc:225] Handling call
0112 08:57:10.622343 (+2348075us) ts_tablet_manager.cc:1145] Deleting tablet c4fc4d2d0eaa47f6a02f518828253057
0112 08:57:10.622402 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.622794 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347518}
I20250112 08:57:10.624140 22180 tablet_service.cc:1467] Processing CreateTablet for tablet 8509c81d4fd54b5dbca4633433f2eb48 (DEFAULT_TABLE table=default.Table_1871a9c14e194ccfae3a16776b48ee2e [id=186484f3ebf14b3193d2baec31cdb49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:10.627482 22458 tablet_bootstrap.cc:654] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.642230 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 205) took 2193 ms. Trace:
I20250112 08:57:10.642750 23103 rpcz_store.cc:276] 0112 08:57:08.448661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.468869 (+ 20208us) service_pool.cc:225] Handling call
0112 08:57:10.641725 (+2172856us) ts_tablet_manager.cc:1145] Deleting tablet 1ad1f85596894fe9992ec4ad115e6928
0112 08:57:10.641800 (+ 75us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.642169 (+ 369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154674}
I20250112 08:57:10.644052 23103 tablet_replica.cc:331] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:10.644966 23103 raft_consensus.cc:2238] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:10.645506 23103 raft_consensus.cc:2267] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:10.647603 23103 ts_tablet_manager.cc:1905] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:10.655484 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:10.661006 22461 tablet_bootstrap.cc:654] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.670696 22565 ts_tablet_manager.cc:1918] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:10.671031 22565 log.cc:1198] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/1514074a19694aa0a42b968a29904c30
I20250112 08:57:10.677706 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44cad3bd5a84e75a41733d8d2f345a8" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:10.678277 22355 raft_consensus.cc:3054] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:10.679486 22565 ts_tablet_manager.cc:1939] T 1514074a19694aa0a42b968a29904c30 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:10.673211 22458 tablet_bootstrap.cc:492] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:10.683102 22458 ts_tablet_manager.cc:1397] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.005s
W20250112 08:57:10.684840 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 119055us
I20250112 08:57:10.685420 22458 raft_consensus.cc:357] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.686080 22458 raft_consensus.cc:383] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.685814 22355 raft_consensus.cc:2463] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:10.686331 22458 raft_consensus.cc:738] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.687259 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 176) took 4975 ms. Trace:
I20250112 08:57:10.687423 22002 leader_election.cc:304] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:10.688447 23234 raft_consensus.cc:2798] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:10.690037 23234 raft_consensus.cc:695] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:10.690973 23234 consensus_queue.cc:237] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:10.688472 22458 consensus_queue.cc:260] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:10.695513 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 80c6ea87589249739461e629eee5f40b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80c6ea87589249739461e629eee5f40b already in progress: creating tablet
I20250112 08:57:10.697079 22458 ts_tablet_manager.cc:1428] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
W20250112 08:57:10.700795 22290 leader_election.cc:343] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 2e62435dce1441a3b2fee50518662d48
I20250112 08:57:10.701176 22290 leader_election.cc:304] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:10.701898 23188 raft_consensus.cc:2743] T 2e62435dce1441a3b2fee50518662d48 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.675592 23131 leader_election.cc:290] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805)
I20250112 08:57:10.687580 22565 rpcz_store.cc:276] 0112 08:57:05.711850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.714414 (+ 2564us) service_pool.cc:225] Handling call
0112 08:57:10.463290 (+4748876us) ts_tablet_manager.cc:1145] Deleting tablet 1514074a19694aa0a42b968a29904c30
0112 08:57:10.463324 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.620771 (+157447us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.670472 (+ 49701us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:10.681438 (+ 10966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4701601}
I20250112 08:57:10.704286 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 177) took 4990 ms. Trace:
I20250112 08:57:10.705812 22458 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:10.704658 22565 rpcz_store.cc:276] 0112 08:57:05.713799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.716854 (+ 3055us) service_pool.cc:225] Handling call
0112 08:57:10.703873 (+4987019us) ts_tablet_manager.cc:1145] Deleting tablet da16d10565a74243a9ab8c5264bf7b1a
0112 08:57:10.703947 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.704231 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4941966}
I20250112 08:57:10.709851 22565 tablet_replica.cc:331] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:10.710593 22565 raft_consensus.cc:2238] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:10.712828 22456 tablet_bootstrap.cc:492] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:10.712158 22565 raft_consensus.cc:2267] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:10.717698 22565 ts_tablet_manager.cc:1905] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:10.719002 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:10.719656 22210 raft_consensus.cc:2463] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:10.724138 22180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8509c81d4fd54b5dbca4633433f2eb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.733556 22461 tablet_bootstrap.cc:492] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:10.733955 22461 ts_tablet_manager.cc:1397] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.000s
I20250112 08:57:10.741170 22456 tablet_bootstrap.cc:654] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.749431 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575891fa458d4d8391d493dbe34fc0b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:10.755079 22861 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:50913 (request call id 28) took 779 ms (client timeout 1000 ms). Trace:
I20250112 08:57:10.756210 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340fcbf7e3cf4038b5bed1b188da18dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.760931 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b164f5284a4484db5b03af09ec68276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:10.767004 23196 raft_consensus.cc:491] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.767470 23196 raft_consensus.cc:513] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:10.769500 23196 leader_election.cc:290] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:57:10.770105 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 71846.4us
I20250112 08:57:10.769503 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:10.773506 22211 raft_consensus.cc:2388] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 53068c99960647728141706e9c50b870 in this term.
W20250112 08:57:10.775597 22861 rpcz_store.cc:269] 0112 08:57:09.976044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.976177 (+ 133us) service_pool.cc:225] Handling call
0112 08:57:10.752599 (+776422us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b5400040454. stack: 00007fbe19bfa853 00007fbe19bfa750 00007fbe1976f584 00000000004fcd24 00000000004fccb9 00007fbe20afad50 00007fbe20b7df2a 00007fbe22822dec 00007fbe2291adee 00007fbe20678fc4 00007fbe20678f52 00007fbe20678e81 00007fbe20678dfc 00007fbe206780b2 00007fbe1e942dfc 00007fbe1e942236
0112 08:57:10.754982 (+ 2383us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1781419264}
I20250112 08:57:10.755635 23223 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:10.776737 23223 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:10.779008 23223 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:10.780977 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:10.781401 23188 raft_consensus.cc:491] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:10.782989 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 335 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:10.786104 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:57:10.788178 23223 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:10.788904 22863 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:40053 (request call id 29) took 901 ms (client timeout 1000 ms). Trace:
W20250112 08:57:10.789230 22863 rpcz_store.cc:269] 0112 08:57:09.887550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.887701 (+ 151us) service_pool.cc:225] Handling call
0112 08:57:10.764346 (+876645us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b5400040454. stack: 00007fbe19bfa853 00007fbe19bfa750 00007fbe1976f584 00000000004fcd24 00000000004fccb9 00007fbe20afad50 00007fbe20b7df2a 00007fbe22822dec 00007fbe2291adee 00007fbe20678fc4 00007fbe20678f52 00007fbe20678e81 00007fbe20678dfc 00007fbe206780b2 00007fbe1e942dfc 00007fbe1e942236
0112 08:57:10.788848 (+ 24502us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1984083200}
I20250112 08:57:10.795802 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:10.797142 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 1 got 0
I20250112 08:57:10.797782 21712 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:10.809274 21711 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:10.810292 21711 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:10.820415 21711 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:10.832208 22461 raft_consensus.cc:357] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.832994 22461 raft_consensus.cc:383] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.833253 22461 raft_consensus.cc:738] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:10.833864 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:10.839715 22456 tablet_bootstrap.cc:492] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:10.840130 22456 ts_tablet_manager.cc:1397] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.127s user 0.008s sys 0.005s
I20250112 08:57:10.841758 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 1514074a19694aa0a42b968a29904c30 (table default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]) successfully deleted
W20250112 08:57:10.842476 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet da16d10565a74243a9ab8c5264bf7b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da16d10565a74243a9ab8c5264bf7b1a already in progress: creating tablet
I20250112 08:57:10.842219 22456 raft_consensus.cc:357] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:10.842820 22456 raft_consensus.cc:383] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:10.843060 22456 raft_consensus.cc:738] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:57:10.843839 21701 catalog_manager.cc:4852] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet c49ec037c22a42108ecb48bc8270191f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49ec037c22a42108ecb48bc8270191f already in progress: creating tablet
I20250112 08:57:10.855420 22456 consensus_queue.cc:260] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:10.856601 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:10.857630 21709 catalog_manager.cc:5526] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:10.858929 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:10.859583 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet c9848b879a2e40aeb840924261e6a100 (table default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe]). Expected version 1 got 0
I20250112 08:57:10.860285 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet a5959caa4d39420981e3b1994df4cb17 (table default.Table_275569887e7c49dd9608ddf9efc7fb39 [id=8172f1c5f3b54191aaa2322ccd8c638d]). Expected version 1 got 0
I20250112 08:57:10.861155 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:10.862056 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
W20250112 08:57:10.718309 21999 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a44cad3bd5a84e75a41733d8d2f345a8
W20250112 08:57:10.862772 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 144492us
I20250112 08:57:10.863231 22456 ts_tablet_manager.cc:1428] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
W20250112 08:57:10.863724 21999 leader_election.cc:343] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:10.864064 22456 tablet_bootstrap.cc:492] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:10.771291 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc82026fcd1541418176a015ea3253cf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:10.866473 22146 leader_election.cc:343] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:10.873435 22456 tablet_bootstrap.cc:654] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.850625 22458 tablet_bootstrap.cc:654] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:10.781819 23188 raft_consensus.cc:513] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.923307 23188 leader_election.cc:290] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:10.933796 22461 consensus_queue.cc:260] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:10.936075 22461 ts_tablet_manager.cc:1428] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.202s user 0.007s sys 0.001s
I20250112 08:57:10.936959 22461 tablet_bootstrap.cc:492] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:10.943692 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 207286us
W20250112 08:57:10.944317 22290 leader_election.cc:343] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 80c6ea87589249739461e629eee5f40b
I20250112 08:57:10.944695 22290 leader_election.cc:304] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:10.968240 23188 raft_consensus.cc:2743] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:10.978528 22290 leader_election.cc:304] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:10.987862 23232 tablet.cc:1721] T 1c1c634513e14e818601894834c41378 P 76718d1596134c83900541c6d398d59e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 08:57:10.990448 23193 tablet.cc:1721] T 1c1c634513e14e818601894834c41378 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 08:57:10.994588 22458 tablet_bootstrap.cc:492] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:10.995040 22458 ts_tablet_manager.cc:1397] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.289s user 0.007s sys 0.003s
I20250112 08:57:10.997465 22458 raft_consensus.cc:357] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.019371 22456 tablet_bootstrap.cc:492] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.019760 22456 ts_tablet_manager.cc:1397] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.156s user 0.032s sys 0.055s
I20250112 08:57:11.022171 22456 raft_consensus.cc:357] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.022795 22456 raft_consensus.cc:383] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.023056 22456 raft_consensus.cc:738] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.023665 22456 consensus_queue.cc:260] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.026471 22456 ts_tablet_manager.cc:1428] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:11.002074 23188 raft_consensus.cc:2798] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:11.032279 23188 raft_consensus.cc:491] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:11.032714 23188 raft_consensus.cc:3054] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.044766 21716 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 1 got 0
I20250112 08:57:11.045362 21716 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 815d246b7d414aadbb69ac692a5d40c7 (table default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb]). Expected version 1 got 0
I20250112 08:57:11.046186 21716 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:11.046671 21716 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:11.047580 21716 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:11.051682 22456 tablet_bootstrap.cc:492] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:11.057689 22456 tablet_bootstrap.cc:654] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.059042 23225 tablet.cc:1721] T 1c1c634513e14e818601894834c41378 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250112 08:57:11.067970 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 89504.7us
I20250112 08:57:11.068498 22456 tablet_bootstrap.cc:492] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.068924 22456 ts_tablet_manager.cc:1397] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20250112 08:57:11.072947 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:11.074543 22323 tablet_service.cc:1467] Processing CreateTablet for tablet ffa0f45a8ee94ad69b403509cb532ed3 (DEFAULT_TABLE table=default.Table_6280d02d39bc47eba2925dcc1fef5835 [id=4c44ec3226a44bef89713c95fed68652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.076527 22327 tablet_service.cc:1467] Processing CreateTablet for tablet 5a162200945646729e1744cd6fd407c3 (DEFAULT_TABLE table=default.Table_4198997d702f4c4bb04a3082456851c8 [id=f6593a616557410586bf02e03726f0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.076922 22326 tablet_service.cc:1467] Processing CreateTablet for tablet 13990acd16404d119298d020b6d14a4e (DEFAULT_TABLE table=default.Table_ba481b6297ae48a4b20921031aef6276 [id=a991cdf15fe34001a629f0a82c393d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.078254 22325 tablet_service.cc:1467] Processing CreateTablet for tablet e65ae6f4959b4a57a7357e1040f8dec8 (DEFAULT_TABLE table=default.Table_fcb376f8af89430c84a3f2c0fea366b1 [id=1ab96aa394664871bf9cf220d2390d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.079572 22324 tablet_service.cc:1467] Processing CreateTablet for tablet 8d28a29fb18843c79df8f58743c6b0b0 (DEFAULT_TABLE table=default.Table_2055fd4bf7224371ba88109fd2e0f629 [id=c0f115b088be4be8b5a6cbba28785a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.071025 22456 raft_consensus.cc:357] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.081199 22456 raft_consensus.cc:383] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.081512 22456 raft_consensus.cc:738] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.082121 22456 consensus_queue.cc:260] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.084203 22456 ts_tablet_manager.cc:1428] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250112 08:57:11.090727 22456 tablet_bootstrap.cc:492] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:10.967857 22461 tablet_bootstrap.cc:654] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.019863 23234 raft_consensus.cc:491] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.092195 23234 raft_consensus.cc:513] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.094120 23234 leader_election.cc:290] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.095887 22456 tablet_bootstrap.cc:654] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.096405 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.098207 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53167910ab0a4accb1d1615d537ba7ae" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:11.098970 22211 raft_consensus.cc:2388] T 53167910ab0a4accb1d1615d537ba7ae P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:11.099956 21999 leader_election.cc:304] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:11.101006 23234 raft_consensus.cc:3054] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.104774 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa0f45a8ee94ad69b403509cb532ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.108624 22327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a162200945646729e1744cd6fd407c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.112017 22458 raft_consensus.cc:383] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.112320 22458 raft_consensus.cc:738] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.112435 22326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13990acd16404d119298d020b6d14a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.112926 22458 consensus_queue.cc:260] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.114838 22456 tablet_bootstrap.cc:492] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.115000 22458 ts_tablet_manager.cc:1428] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.120s user 0.003s sys 0.000s
I20250112 08:57:11.115238 22456 ts_tablet_manager.cc:1397] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20250112 08:57:11.116503 22325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65ae6f4959b4a57a7357e1040f8dec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.120396 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d28a29fb18843c79df8f58743c6b0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.121299 22456 raft_consensus.cc:357] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.121928 22456 raft_consensus.cc:383] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.122145 22456 raft_consensus.cc:738] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.122701 22456 consensus_queue.cc:260] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.123302 22458 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:11.127897 23234 raft_consensus.cc:2743] T 53167910ab0a4accb1d1615d537ba7ae P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:11.143502 22565 ts_tablet_manager.cc:1918] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:11.143992 22565 log.cc:1198] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:11.144781 22565 ts_tablet_manager.cc:1939] T 35f60d1f4bed4721b73deb9ba787417c P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
W20250112 08:57:11.145200 21702 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 48adce00f3644192880a8b16b5e514d9 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 48adce00f3644192880a8b16b5e514d9 consensus state has no leader
I20250112 08:57:11.153126 22456 ts_tablet_manager.cc:1428] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20250112 08:57:11.165558 22456 tablet_bootstrap.cc:492] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:11.172050 22456 tablet_bootstrap.cc:654] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.185985 22456 tablet_bootstrap.cc:492] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.186368 22456 ts_tablet_manager.cc:1397] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20250112 08:57:11.187413 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 179) took 5387 ms. Trace:
I20250112 08:57:11.195008 22565 rpcz_store.cc:276] 0112 08:57:05.800304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.803305 (+ 3001us) service_pool.cc:225] Handling call
0112 08:57:10.709712 (+4906407us) ts_tablet_manager.cc:1145] Deleting tablet 35f60d1f4bed4721b73deb9ba787417c
0112 08:57:10.709745 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.976528 (+266783us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.143296 (+166768us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.187351 (+ 44055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4901880}
I20250112 08:57:11.196281 22458 tablet_bootstrap.cc:654] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.198683 23196 raft_consensus.cc:491] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.199179 23196 raft_consensus.cc:513] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.199970 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 180) took 5395 ms. Trace:
I20250112 08:57:11.200179 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 35f60d1f4bed4721b73deb9ba787417c (table default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]) successfully deleted
I20250112 08:57:11.200282 22565 rpcz_store.cc:276] 0112 08:57:05.801616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:05.807964 (+ 6348us) service_pool.cc:225] Handling call
0112 08:57:11.196943 (+5388979us) ts_tablet_manager.cc:1145] Deleting tablet 8c934e1a62444e64911a279f6a544c96
0112 08:57:11.196984 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.197129 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5386584}
I20250112 08:57:11.201053 23196 leader_election.cc:290] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.188663 22456 raft_consensus.cc:357] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.203135 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163d4e4a20cd4f0e80e559fdd04343b4" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:11.203933 22456 raft_consensus.cc:383] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.204181 22456 raft_consensus.cc:738] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
W20250112 08:57:11.204496 22146 leader_election.cc:343] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.205454 22456 consensus_queue.cc:260] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:57:11.208180 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 8c934e1a62444e64911a279f6a544c96 because the tablet was not found. No further retry: Not found: Tablet not found: 8c934e1a62444e64911a279f6a544c96
I20250112 08:57:11.213630 23103 ts_tablet_manager.cc:1918] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:11.214022 23103 log.cc:1198] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/1514074a19694aa0a42b968a29904c30
I20250112 08:57:11.214668 23103 ts_tablet_manager.cc:1939] T 1514074a19694aa0a42b968a29904c30 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:11.214706 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 185) took 5080 ms. Trace:
I20250112 08:57:11.215113 22565 rpcz_store.cc:276] 0112 08:57:06.134271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.136488 (+ 2217us) service_pool.cc:225] Handling call
0112 08:57:11.214349 (+5077861us) ts_tablet_manager.cc:1145] Deleting tablet ea15d9405b9944db8e00121965b9ffde
0112 08:57:11.214393 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.214650 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5059679}
I20250112 08:57:11.219173 22456 ts_tablet_manager.cc:1428] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.033s user 0.003s sys 0.003s
I20250112 08:57:11.220003 22456 tablet_bootstrap.cc:492] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:57:11.220045 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet ea15d9405b9944db8e00121965b9ffde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea15d9405b9944db8e00121965b9ffde already in progress: creating tablet
I20250112 08:57:11.221014 22565 tablet_replica.cc:331] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:11.224519 22565 raft_consensus.cc:2238] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:11.228127 22565 raft_consensus.cc:2267] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:11.230131 22565 ts_tablet_manager.cc:1905] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:11.237480 22456 tablet_bootstrap.cc:654] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.232081 22458 tablet_bootstrap.cc:492] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:11.238052 22458 ts_tablet_manager.cc:1397] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.003s
I20250112 08:57:11.240319 22458 raft_consensus.cc:357] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.242524 22456 tablet_bootstrap.cc:492] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.242584 22458 raft_consensus.cc:383] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.242875 22456 ts_tablet_manager.cc:1397] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20250112 08:57:11.244864 22456 raft_consensus.cc:357] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.242925 22458 raft_consensus.cc:738] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.245496 22456 raft_consensus.cc:383] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.245273 22330 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18dfd51dbe40f9830c9f873347375d (DEFAULT_TABLE table=default.Table_33acf77ade0f4b5d8e348dfe6e38b4fa [id=f6ce4a5bdd694fc7bdda81f7e020e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.246294 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18dfd51dbe40f9830c9f873347375d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.245714 22456 raft_consensus.cc:738] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.247277 22456 consensus_queue.cc:260] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.248302 22458 consensus_queue.cc:260] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.253300 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 221796ee0715492eaf4c50619e55e2fc (DEFAULT_TABLE table=default.Table_9b44fe9ca083478fa41484d20bf2a49d [id=e021f84a29554f89a70beb984932c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.254341 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221796ee0715492eaf4c50619e55e2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.249390 23234 raft_consensus.cc:491] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.254680 23234 raft_consensus.cc:513] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.256620 23234 leader_election.cc:290] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.258872 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.259581 22355 raft_consensus.cc:2388] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:11.260396 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:11.261001 22211 raft_consensus.cc:2463] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:11.261893 22456 ts_tablet_manager.cc:1428] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250112 08:57:11.262240 21999 leader_election.cc:304] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters:
I20250112 08:57:11.262457 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa44c174012470d80fd00970bb27883 (DEFAULT_TABLE table=default.Table_b639de28dc9642f29a1f0caeecc05a57 [id=d15d4c0cf8e940a7a857afa2ae304749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.262758 22456 tablet_bootstrap.cc:492] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:11.263415 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa44c174012470d80fd00970bb27883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.263955 23234 raft_consensus.cc:2798] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:11.264276 23234 raft_consensus.cc:491] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:11.264581 23234 raft_consensus.cc:3054] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.268290 22456 tablet_bootstrap.cc:654] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.268782 22337 tablet_service.cc:1467] Processing CreateTablet for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8 (DEFAULT_TABLE table=default.Table_6872481b576147d695568e3d5bf35e77 [id=321f8586dfc440ff9262f12bbf0e882a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.269840 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8127d5d8b1a4fa5b2e860c92b6ebfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.272873 23234 raft_consensus.cc:513] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.275465 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:11.276141 22355 raft_consensus.cc:2388] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:11.278443 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:11.278971 22211 raft_consensus.cc:3054] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.282914 23196 raft_consensus.cc:491] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.283345 23196 raft_consensus.cc:513] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.285086 23196 leader_election.cc:290] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.285703 22211 raft_consensus.cc:2463] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:11.286095 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deac41b2a0584528b048e2a0ef09485e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:11.286799 21999 leader_election.cc:304] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e; no voters: 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:11.287345 22146 leader_election.cc:343] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.287528 23131 raft_consensus.cc:2798] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:11.287928 23131 raft_consensus.cc:695] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:11.288661 23131 consensus_queue.cc:237] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.312724 22458 ts_tablet_manager.cc:1428] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250112 08:57:11.316411 22458 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:11.317065 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 38baf231bceb4591b3d3e86006542f3a (DEFAULT_TABLE table=default.Table_44de87cd557a4c878db8d37210254226 [id=2a955059b25e440d861c5635c041a8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.320298 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38baf231bceb4591b3d3e86006542f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.332005 23234 leader_election.cc:290] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.344316 22461 tablet_bootstrap.cc:492] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:11.344825 22461 ts_tablet_manager.cc:1397] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.408s user 0.006s sys 0.003s
I20250112 08:57:11.334362 23188 raft_consensus.cc:513] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.353492 22336 tablet_service.cc:1467] Processing CreateTablet for tablet cde9fbb824d040ca90b4109703790810 (DEFAULT_TABLE table=default.Table_00c8e2048c444eeca4149e2b643a7abe [id=a5a69fa5f6df4391ac5c0ee7288668cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.354521 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde9fbb824d040ca90b4109703790810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.356504 22461 raft_consensus.cc:357] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.357172 22461 raft_consensus.cc:383] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.357424 22461 raft_consensus.cc:738] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.358035 22461 consensus_queue.cc:260] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.392489 22458 tablet_bootstrap.cc:654] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.393663 22337 tablet_service.cc:1467] Processing CreateTablet for tablet 1722022567174f7a8cc9c85fe6655b4c (DEFAULT_TABLE table=default.Table_6ba47fafc72c4861b7adf19ac9fba3cf [id=c6661e6ba3d64dc7af7c461c79b4934f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.394690 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1722022567174f7a8cc9c85fe6655b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.402729 23188 leader_election.cc:290] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.403731 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 206) took 2404 ms. Trace:
I20250112 08:57:11.404174 23103 rpcz_store.cc:276] 0112 08:57:08.999013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.999125 (+ 112us) service_pool.cc:225] Handling call
0112 08:57:10.643827 (+1644702us) ts_tablet_manager.cc:1145] Deleting tablet 1514074a19694aa0a42b968a29904c30
0112 08:57:10.643880 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:10.956315 (+312435us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.213381 (+257066us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.403678 (+190297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1644037}
I20250112 08:57:11.405540 23103 tablet_replica.cc:331] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:11.417059 22338 tablet_service.cc:1467] Processing CreateTablet for tablet ae66efbd303d4a25b4c547915109bca7 (DEFAULT_TABLE table=default.Table_b36df5002f114aceb067f4c79bc6c9ff [id=a7f5894f46c143948abb4fe47ba94059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.418733 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5a16774d8e4bee8ae6cbdc760d003a (DEFAULT_TABLE table=default.Table_79dac8e9ea5d483397bd1ec1cbc74403 [id=dad0bc3b967c4dec94f065c3d35df670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.420423 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 3b164f5284a4484db5b03af09ec68276 (DEFAULT_TABLE table=default.Table_660015f62b6c4db59a358f76526c5dad [id=f55e906f67104045afdabcb6b80c4345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.421679 22338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae66efbd303d4a25b4c547915109bca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.425145 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b164f5284a4484db5b03af09ec68276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.428136 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d5a16774d8e4bee8ae6cbdc760d003a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.456737 22458 tablet_bootstrap.cc:492] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:11.457122 22458 ts_tablet_manager.cc:1397] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.143s user 0.006s sys 0.002s
I20250112 08:57:11.457849 22325 tablet_service.cc:1467] Processing CreateTablet for tablet 137e2275af51421cac4ca629dfa809ba (DEFAULT_TABLE table=default.Table_60d957dcf38841e5a1ab61b409df19af [id=3390616d770745a39ee27f66c84ce788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.459102 22325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 137e2275af51421cac4ca629dfa809ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.465830 22324 tablet_service.cc:1467] Processing CreateTablet for tablet 8509c81d4fd54b5dbca4633433f2eb48 (DEFAULT_TABLE table=default.Table_1871a9c14e194ccfae3a16776b48ee2e [id=186484f3ebf14b3193d2baec31cdb49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.466970 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8509c81d4fd54b5dbca4633433f2eb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.471513 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
W20250112 08:57:11.472347 21999 consensus_peers.cc:487] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a44cad3bd5a84e75a41733d8d2f345a8. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:11.479034 22461 ts_tablet_manager.cc:1428] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.134s user 0.006s sys 0.000s
I20250112 08:57:11.479012 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 422e1484cd84479abba824cc8a6fae1f (table default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]) successfully deleted
I20250112 08:57:11.479174 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93c5e0d2a14c16a8080b21da64eb85" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:11.479821 22213 raft_consensus.cc:2388] T 7d93c5e0d2a14c16a8080b21da64eb85 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c608785d6874abf9cc28db11a979ddd in current term 1: Already voted for candidate 76718d1596134c83900541c6d398d59e in this term.
I20250112 08:57:11.481945 22458 raft_consensus.cc:357] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.482618 22458 raft_consensus.cc:383] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.482877 22458 raft_consensus.cc:738] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.486104 22565 ts_tablet_manager.cc:1918] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:11.486907 22565 log.cc:1198] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:11.487546 22565 ts_tablet_manager.cc:1939] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:11.494751 22456 tablet_bootstrap.cc:492] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.495172 22456 ts_tablet_manager.cc:1397] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.233s user 0.079s sys 0.116s
W20250112 08:57:11.507706 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet c4fc4d2d0eaa47f6a02f518828253057 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4fc4d2d0eaa47f6a02f518828253057 already in progress: creating tablet
W20250112 08:57:11.508433 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 1ad1f85596894fe9992ec4ad115e6928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad1f85596894fe9992ec4ad115e6928 already in progress: creating tablet
I20250112 08:57:11.508921 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:11.511169 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:11.511637 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:11.511763 22211 raft_consensus.cc:2463] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:11.513610 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 1514074a19694aa0a42b968a29904c30 (table default.Table_1acedaae5cd2449d9d9cbc5a91994176 [id=aa98800c01dc46c59a6050b60d787831]) successfully deleted
I20250112 08:57:11.514731 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3e34e1d70a4c63956a5be660e02b21" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:11.515220 22211 raft_consensus.cc:3054] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.526643 22456 raft_consensus.cc:357] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.527347 22456 raft_consensus.cc:383] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.541260 22456 raft_consensus.cc:738] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.541910 22456 consensus_queue.cc:260] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.550141 23196 raft_consensus.cc:491] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.550595 23196 raft_consensus.cc:513] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.552408 23196 leader_election.cc:290] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.554123 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153bb666e5d5440ea22d59260a8299c6" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:11.555408 22146 leader_election.cc:343] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.561167 23196 raft_consensus.cc:491] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.561559 23196 raft_consensus.cc:513] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.564610 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c48b95e2f24f8a8a2638d4df8ba3b8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:11.565855 22146 leader_election.cc:343] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.566365 23196 leader_election.cc:290] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.572675 22211 raft_consensus.cc:2463] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:11.573257 22456 ts_tablet_manager.cc:1428] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.078s user 0.007s sys 0.001s
I20250112 08:57:11.580060 23264 raft_consensus.cc:491] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.580484 23264 raft_consensus.cc:513] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.560801 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet c49ec037c22a42108ecb48bc8270191f with delete_type TABLET_DATA_DELETED (replaced by tablet b7c3c7162ff144a79715944b7c9f0371) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:11.582348 23264 leader_election.cc:290] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.583274 22456 tablet_bootstrap.cc:492] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:11.583345 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3fbe9ad4224e5e8904c1b7dc7e231c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:11.584532 22146 leader_election.cc:343] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.588613 23103 raft_consensus.cc:2238] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:11.591148 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 340fcbf7e3cf4038b5bed1b188da18dd (DEFAULT_TABLE table=default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.593421 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 186) took 5438 ms. Trace:
I20250112 08:57:11.593765 22565 rpcz_store.cc:276] 0112 08:57:06.155328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.156115 (+ 787us) service_pool.cc:225] Handling call
0112 08:57:11.220873 (+5064758us) ts_tablet_manager.cc:1145] Deleting tablet ac1889a1ab4f4d8aa4f7915d91f2cf27
0112 08:57:11.220913 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.349029 (+128116us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.463299 (+114270us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.593347 (+130048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5045867}
I20250112 08:57:11.595064 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (table default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]) successfully deleted
I20250112 08:57:11.596156 23234 consensus_queue.cc:1035] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:11.607033 23103 raft_consensus.cc:2267] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:11.608866 22456 tablet_bootstrap.cc:654] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.612676 22565 tablet_replica.cc:331] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:11.619567 22565 raft_consensus.cc:2238] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:11.620852 22565 raft_consensus.cc:2267] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:11.624238 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 575891fa458d4d8391d493dbe34fc0b3 (DEFAULT_TABLE table=default.Table_55a599c370f441969aad1da2ebfb4793 [id=14d632c40d984bd3aa1930f930705313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.625782 23103 ts_tablet_manager.cc:1905] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:11.637305 22565 ts_tablet_manager.cc:1905] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:11.483526 22458 consensus_queue.cc:260] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.654073 22458 ts_tablet_manager.cc:1428] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.197s user 0.005s sys 0.001s
I20250112 08:57:11.654184 23271 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:11.658267 23271 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:11.667829 22458 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:11.709146 23271 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:11.709933 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 343 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:11.711930 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 343 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:11.714972 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
I20250112 08:57:11.720606 23271 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:11.728266 22327 consensus_meta.cc:287] T 5a162200945646729e1744cd6fd407c3 P 8c608785d6874abf9cc28db11a979ddd: Time spent flushing consensus metadata: real 0.521s user 0.007s sys 0.000s
I20250112 08:57:11.749531 22461 tablet_bootstrap.cc:492] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:11.754452 22565 ts_tablet_manager.cc:1918] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:11.754812 22565 log.cc:1198] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/ef66f5e785864d1c845f5c5a5f944473
I20250112 08:57:11.755550 22565 ts_tablet_manager.cc:1939] T ef66f5e785864d1c845f5c5a5f944473 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:11.772049 22456 tablet_bootstrap.cc:492] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.772459 22456 ts_tablet_manager.cc:1397] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.199s user 0.024s sys 0.019s
I20250112 08:57:11.774991 22461 tablet_bootstrap.cc:654] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.774705 22456 raft_consensus.cc:357] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.775347 22456 raft_consensus.cc:383] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.775604 22456 raft_consensus.cc:738] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.776815 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 340fcbf7e3cf4038b5bed1b188da18dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.796443 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575891fa458d4d8391d493dbe34fc0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.796160 22456 consensus_queue.cc:260] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.809305 22456 ts_tablet_manager.cc:1428] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250112 08:57:11.810209 22456 tablet_bootstrap.cc:492] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:11.819386 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61e80e259db4645942141754c8491a3" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.821029 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:11.821645 22355 raft_consensus.cc:3054] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:11.835343 22456 tablet_bootstrap.cc:654] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:11.837239 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc82026fcd1541418176a015ea3253cf" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.837430 22355 raft_consensus.cc:2463] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:11.839159 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "163d4e4a20cd4f0e80e559fdd04343b4" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.845606 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deac41b2a0584528b048e2a0ef09485e" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.850873 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153bb666e5d5440ea22d59260a8299c6" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.852695 23268 raft_consensus.cc:491] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.853214 23268 raft_consensus.cc:513] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.855536 23268 leader_election.cc:290] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.868012 23264 raft_consensus.cc:491] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.868484 23264 raft_consensus.cc:513] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.869518 22461 tablet_bootstrap.cc:492] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:11.869911 22461 ts_tablet_manager.cc:1397] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.003s
I20250112 08:57:11.870383 23264 leader_election.cc:290] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.871727 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 2045cc7626714c63baaf0325600edf54 (DEFAULT_TABLE table=default.Table_f3931a3aec8748a2987f29762a28d078 [id=03b859c1412f4e5f9da6dde2cd305bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.872728 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045cc7626714c63baaf0325600edf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.872257 22461 raft_consensus.cc:357] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.872987 22461 raft_consensus.cc:383] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.873239 22461 raft_consensus.cc:738] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.873864 22461 consensus_queue.cc:260] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.875041 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:11.876250 22461 ts_tablet_manager.cc:1428] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250112 08:57:11.876332 23264 raft_consensus.cc:491] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.877126 22461 tablet_bootstrap.cc:492] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:11.876749 23264 raft_consensus.cc:513] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.879562 23264 leader_election.cc:290] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.883383 22036 tablet_service.cc:1467] Processing CreateTablet for tablet 2045cc7626714c63baaf0325600edf54 (DEFAULT_TABLE table=default.Table_f3931a3aec8748a2987f29762a28d078 [id=03b859c1412f4e5f9da6dde2cd305bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.881695 22184 tablet_service.cc:1467] Processing CreateTablet for tablet f3743f45c75f4cc682f50eebda212633 (DEFAULT_TABLE table=default.Table_e24a40491b8f455eb04b0a4e567df5d5 [id=2f602150372549818fb7e4787b8a823d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.885078 22184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3743f45c75f4cc682f50eebda212633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.891773 22456 tablet_bootstrap.cc:492] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:11.892170 22456 ts_tablet_manager.cc:1397] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.000s
I20250112 08:57:11.896881 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a8864649341c889d805c5599c12af (DEFAULT_TABLE table=default.Table_03c9aad903a742bb96dd314764735ea0 [id=17cbe93491d6453682dc84a7e657bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.897817 22193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2a8864649341c889d805c5599c12af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.895169 22456 raft_consensus.cc:357] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.899003 22456 raft_consensus.cc:383] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:11.899263 22456 raft_consensus.cc:738] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:11.899869 22456 consensus_queue.cc:260] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.902791 23196 raft_consensus.cc:491] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.903200 23196 raft_consensus.cc:513] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.904896 23196 leader_election.cc:290] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.897245 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:11.913259 21708 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:11.915297 22180 tablet_service.cc:1467] Processing CreateTablet for tablet b7c3c7162ff144a79715944b7c9f0371 (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.916435 22180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c3c7162ff144a79715944b7c9f0371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.920670 22036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045cc7626714c63baaf0325600edf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.928043 23264 raft_consensus.cc:491] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.928463 23264 raft_consensus.cc:513] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:11.930966 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 188) took 4973 ms. Trace:
I20250112 08:57:11.931135 23264 leader_election.cc:290] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:11.931279 22565 rpcz_store.cc:276] 0112 08:57:06.957408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:06.957546 (+ 138us) service_pool.cc:225] Handling call
0112 08:57:11.612518 (+4654972us) ts_tablet_manager.cc:1145] Deleting tablet ef66f5e785864d1c845f5c5a5f944473
0112 08:57:11.612562 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.717111 (+104549us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.754215 (+ 37104us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.930904 (+176689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4654436}
I20250112 08:57:11.932448 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 192) took 4455 ms. Trace:
I20250112 08:57:11.932706 22565 rpcz_store.cc:276] 0112 08:57:07.477282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.477451 (+ 169us) service_pool.cc:225] Handling call
0112 08:57:11.932113 (+4454662us) ts_tablet_manager.cc:1145] Deleting tablet 1df6282224b849a488071323e2e69e1c
0112 08:57:11.932151 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.932399 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4454138}
I20250112 08:57:11.933470 22456 ts_tablet_manager.cc:1428] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250112 08:57:11.933544 22565 tablet_replica.cc:331] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:11.934194 22565 raft_consensus.cc:2238] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:11.934574 22565 raft_consensus.cc:2267] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:11.840088 23234 raft_consensus.cc:491] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.937000 23234 raft_consensus.cc:513] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.938686 23234 leader_election.cc:290] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:11.941201 22044 tablet_service.cc:1467] Processing CreateTablet for tablet b7c3c7162ff144a79715944b7c9f0371 (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.942179 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c3c7162ff144a79715944b7c9f0371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.942438 22045 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a8864649341c889d805c5599c12af (DEFAULT_TABLE table=default.Table_03c9aad903a742bb96dd314764735ea0 [id=17cbe93491d6453682dc84a7e657bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.943447 22045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2a8864649341c889d805c5599c12af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.941208 22048 tablet_service.cc:1467] Processing CreateTablet for tablet f3743f45c75f4cc682f50eebda212633 (DEFAULT_TABLE table=default.Table_e24a40491b8f455eb04b0a4e567df5d5 [id=2f602150372549818fb7e4787b8a823d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:11.944752 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3743f45c75f4cc682f50eebda212633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:11.884738 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c48b95e2f24f8a8a2638d4df8ba3b8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.886236 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3fbe9ad4224e5e8904c1b7dc7e231c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.886639 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340f93f630a34b3b9a9386999e2684e8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:11.886672 22350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228f04faa5c4f68a95adfb295235047" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
W20250112 08:57:11.887391 22290 leader_election.cc:343] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.927094 23268 raft_consensus.cc:491] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:11.952410 23268 raft_consensus.cc:513] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:11.954327 23268 leader_election.cc:290] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:11.951800 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 64431.4us
W20250112 08:57:11.955272 22290 leader_election.cc:343] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.955685 22290 leader_election.cc:304] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:11.956758 23209 raft_consensus.cc:2743] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:11.957985 22565 ts_tablet_manager.cc:1905] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:11.883881 23103 ts_tablet_manager.cc:1918] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:11.967550 23103 log.cc:1198] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:11.968354 23103 ts_tablet_manager.cc:1939] T 35f60d1f4bed4721b73deb9ba787417c P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
W20250112 08:57:11.976389 22290 leader_election.cc:343] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:11.977016 22290 leader_election.cc:343] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.983003 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 207) took 2983 ms. Trace:
W20250112 08:57:11.987795 22149 leader_election.cc:343] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:11.988319 22149 leader_election.cc:304] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:11.989455 23264 raft_consensus.cc:2743] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:11.941460 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet ef66f5e785864d1c845f5c5a5f944473 (table default.Table_53a046eb23ec4f73b2e96111509abefe [id=ee892ee1b62e438688ede44ab3e85711]) successfully deleted
W20250112 08:57:11.994964 21701 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 53520.4us
W20250112 08:57:11.995792 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 1df6282224b849a488071323e2e69e1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df6282224b849a488071323e2e69e1c already in progress: creating tablet
W20250112 08:57:12.001142 22149 leader_election.cc:343] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.001530 22149 leader_election.cc:304] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.002516 23264 raft_consensus.cc:2743] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.003408 22190 tablet_service.cc:1467] Processing CreateTablet for tablet edf7a555e58c4a58b936e7e03ef6d955 (DEFAULT_TABLE table=default.Table_f3352b4abff244e88e533d442b24437d [id=9e070fa9e42c47adb9f1ae8fe5e194e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.004889 22461 tablet_bootstrap.cc:654] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.006523 22149 leader_election.cc:304] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.007643 23264 raft_consensus.cc:2798] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:12.008114 23264 raft_consensus.cc:695] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:12.008761 22048 tablet_service.cc:1467] Processing CreateTablet for tablet edf7a555e58c4a58b936e7e03ef6d955 (DEFAULT_TABLE table=default.Table_f3352b4abff244e88e533d442b24437d [id=9e070fa9e42c47adb9f1ae8fe5e194e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.008852 23264 consensus_queue.cc:237] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.009951 22048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf7a555e58c4a58b936e7e03ef6d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.013751 22044 tablet_service.cc:1467] Processing CreateTablet for tablet 790ac01251084cca90a7db0be0367fa2 (DEFAULT_TABLE table=default.Table_8597641f7fe6456fb720caf6be070864 [id=170c9bdb5e2c4c3c996505274364cf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.014751 22044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790ac01251084cca90a7db0be0367fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.020682 22181 tablet_service.cc:1467] Processing CreateTablet for tablet 790ac01251084cca90a7db0be0367fa2 (DEFAULT_TABLE table=default.Table_8597641f7fe6456fb720caf6be070864 [id=170c9bdb5e2c4c3c996505274364cf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.021212 22039 tablet_service.cc:1467] Processing CreateTablet for tablet 1868b5cbb9a24196a47d4e8e6066bf78 (DEFAULT_TABLE table=default.Table_9bf70ac0df104d21b87badcbe3f09bf8 [id=f4382d0106df4ad49859bf940699099c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.022060 22039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1868b5cbb9a24196a47d4e8e6066bf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.025046 22456 tablet_bootstrap.cc:492] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870: Bootstrap starting.
W20250112 08:57:12.028648 22149 leader_election.cc:343] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.028851 22190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf7a555e58c4a58b936e7e03ef6d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.029047 22149 leader_election.cc:304] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.029956 22181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790ac01251084cca90a7db0be0367fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.030494 22180 tablet_service.cc:1467] Processing CreateTablet for tablet 1868b5cbb9a24196a47d4e8e6066bf78 (DEFAULT_TABLE table=default.Table_9bf70ac0df104d21b87badcbe3f09bf8 [id=f4382d0106df4ad49859bf940699099c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.031430 22180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1868b5cbb9a24196a47d4e8e6066bf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.032104 23196 raft_consensus.cc:2743] T 163d4e4a20cd4f0e80e559fdd04343b4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.039209 22456 tablet_bootstrap.cc:654] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:12.040236 22149 leader_election.cc:343] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.040665 22149 leader_election.cc:304] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.041410 23196 raft_consensus.cc:2743] T deac41b2a0584528b048e2a0ef09485e P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.042248 22187 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb27e51282465cb03f2f9125afcb5e (DEFAULT_TABLE table=default.Table_214464b532cf4fc997136cca2adb554a [id=1912a594d679412db79ac9996db3cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.043402 22187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89eb27e51282465cb03f2f9125afcb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.044323 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:12.044632 22186 tablet_service.cc:1467] Processing CreateTablet for tablet 504b71c1cdd84c019774a9c9d8ab2b11 (DEFAULT_TABLE table=default.Table_e0510c7c9f7b4a39b4229444845528fd [id=355da5876a5640249a35d1b09fdbce11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.044905 22211 raft_consensus.cc:2463] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:12.045647 22186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504b71c1cdd84c019774a9c9d8ab2b11. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 08:57:12.053470 22149 leader_election.cc:343] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.053859 22149 leader_election.cc:304] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.054555 23196 raft_consensus.cc:2743] T 153bb666e5d5440ea22d59260a8299c6 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:12.068596 22149 leader_election.cc:343] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.068987 22149 leader_election.cc:304] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.069597 23196 raft_consensus.cc:2743] T c7c48b95e2f24f8a8a2638d4df8ba3b8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:12.070524 22149 leader_election.cc:343] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.080976 22037 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb27e51282465cb03f2f9125afcb5e (DEFAULT_TABLE table=default.Table_214464b532cf4fc997136cca2adb554a [id=1912a594d679412db79ac9996db3cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.082337 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:12.082634 22037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89eb27e51282465cb03f2f9125afcb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.082475 22038 tablet_service.cc:1467] Processing CreateTablet for tablet 504b71c1cdd84c019774a9c9d8ab2b11 (DEFAULT_TABLE table=default.Table_e0510c7c9f7b4a39b4229444845528fd [id=355da5876a5640249a35d1b09fdbce11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.083571 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_15a31e5fe0e74322bcbefd51437a08b7 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 1 got 0
I20250112 08:57:11.983397 23103 rpcz_store.cc:276] 0112 08:57:09.233890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.240665 (+ 6775us) service_pool.cc:225] Handling call
0112 08:57:11.405381 (+2164716us) ts_tablet_manager.cc:1145] Deleting tablet 35f60d1f4bed4721b73deb9ba787417c
0112 08:57:11.405431 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:11.606494 (+201063us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b58000c1db0. stack: 00007fe54c963853 00007fe54c963750 00007fe54c4d8584 00000000004fcd24 00000000004fccb9 00007fe5538636d1 00007fe55385a1c3 00007fe5538bdaba 00007fe5538e6375 00007fe553d031ac 00007fe554913d5a 00007fe554943ca0 00007fe55492ae6d 00007fe55492ae19 00007fe55492ada9 00007fe55492ad71
0112 08:57:11.813951 (+207457us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.883654 (+ 69703us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:11.982941 (+ 99287us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86018,"spinlock_wait_cycles":34746112,"tablet-delete.queue_time_us":2164062}
I20250112 08:57:12.087867 22038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504b71c1cdd84c019774a9c9d8ab2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.088408 23103 tablet_replica.cc:331] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:12.089335 23103 raft_consensus.cc:2238] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.089828 23103 pending_rounds.cc:62] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:12.090142 23103 pending_rounds.cc:69] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409444317483008 op_type: NO_OP noop_request { }
I20250112 08:57:12.090672 23103 raft_consensus.cc:2883] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:12.081575 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.100124 23103 raft_consensus.cc:2267] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:12.100415 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:12.101173 22355 raft_consensus.cc:2388] T 98ddac23210248e3be1aba6591641b2d P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76718d1596134c83900541c6d398d59e in current term 1: Already voted for candidate 53068c99960647728141706e9c50b870 in this term.
I20250112 08:57:12.108595 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:12.109781 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:12.070855 22149 leader_election.cc:304] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.110493 21717 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:12.081831 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df6282224b849a488071323e2e69e1c" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.112661 23264 raft_consensus.cc:2743] T 6a3fbe9ad4224e5e8904c1b7dc7e231c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:12.113610 22149 leader_election.cc:343] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:12.114147 22149 leader_election.cc:343] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.117626 22456 tablet_bootstrap.cc:492] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:12.118021 22456 ts_tablet_manager.cc:1397] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.093s user 0.001s sys 0.009s
W20250112 08:57:12.118930 21713 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d686b08555414954aba52ea077889660 consensus state has no leader
I20250112 08:57:12.119449 21713 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56650 (request call id 297) took 2901 ms. Trace:
I20250112 08:57:12.119382 22323 tablet_service.cc:1467] Processing CreateTablet for tablet 2045cc7626714c63baaf0325600edf54 (DEFAULT_TABLE table=default.Table_f3931a3aec8748a2987f29762a28d078 [id=03b859c1412f4e5f9da6dde2cd305bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.120474 22323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045cc7626714c63baaf0325600edf54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.120258 22456 raft_consensus.cc:357] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.120867 22456 raft_consensus.cc:383] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.121100 22456 raft_consensus.cc:738] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.119720 21713 rpcz_store.cc:276] 0112 08:57:09.218520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.278985 (+ 60465us) service_pool.cc:225] Handling call
0112 08:57:09.280002 (+ 1017us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:57:09.294043 (+ 14041us) catalog_manager.cc:3627] Apply alter schema
0112 08:57:09.294573 (+ 530us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 08:57:09.294744 (+ 171us) catalog_manager.cc:3817] Updating metadata on disk
0112 08:57:09.340763 (+ 46019us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:09.340788 (+ 25us) write_op.cc:454] Released schema lock
0112 08:57:09.340986 (+ 198us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 08:57:12.119389 (+2778403us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:09.321595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:09.321919 (+ 324us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:09.321947 (+ 28us) write_op.cc:435] Acquired schema lock
0112 08:57:09.321959 (+ 12us) tablet.cc:659] Decoding operations
0112 08:57:09.322212 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:09.322246 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:09.322265 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:09.322409 (+ 144us) tablet.cc:698] Row locks acquired
0112 08:57:09.322421 (+ 12us) write_op.cc:260] PREPARE: finished
0112 08:57:09.322519 (+ 98us) write_op.cc:270] Start()
0112 08:57:09.322591 (+ 72us) write_op.cc:276] Timestamp: P: 1736672229322475 usec, L: 0
0112 08:57:09.322604 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 08:57:09.323056 (+ 452us) log.cc:844] Serialized 405 byte log entry
0112 08:57:09.334765 (+ 11709us) op_driver.cc:464] REPLICATION: finished
0112 08:57:09.335105 (+ 340us) write_op.cc:301] APPLY: starting
0112 08:57:09.335173 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:09.335270 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:09.335392 (+ 122us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:09.335622 (+ 230us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:09.335779 (+ 157us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:09.335835 (+ 56us) write_op.cc:312] APPLY: finished
0112 08:57:09.336054 (+ 219us) log.cc:844] Serialized 23 byte log entry
0112 08:57:09.336213 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:09.336277 (+ 64us) write_op.cc:454] Released schema lock
0112 08:57:09.336311 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"apply.run_cpu_time_us":1369,"apply.run_wall_time_us":1367,"num_ops":1,"prepare.queue_time_us":25733,"prepare.run_cpu_time_us":1753,"prepare.run_wall_time_us":1754,"raft.queue_time_us":1210,"raft.run_cpu_time_us":615,"raft.run_wall_time_us":615,"replication_time_us":12051}]]}
I20250112 08:57:12.122203 22456 consensus_queue.cc:260] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.126430 21675 master-stress-test.cc:579] Killed test thread 5/10
I20250112 08:57:12.126677 21675 master-stress-test.cc:577] Killing test thread 6/10
I20250112 08:57:12.126890 21675 master-stress-test.cc:579] Killed test thread 6/10
I20250112 08:57:12.127085 21675 master-stress-test.cc:577] Killing test thread 7/10
I20250112 08:57:12.127559 21675 master-stress-test.cc:579] Killed test thread 7/10
I20250112 08:57:12.127754 21675 master-stress-test.cc:577] Killing test thread 8/10
I20250112 08:57:12.130769 22565 ts_tablet_manager.cc:1918] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:12.131145 22565 log.cc:1198] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/267ab3e5955b4d3bbeca44baf021b328
I20250112 08:57:12.132150 22461 tablet_bootstrap.cc:492] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:12.132560 22461 ts_tablet_manager.cc:1397] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.256s user 0.017s sys 0.002s
I20250112 08:57:12.134908 22461 raft_consensus.cc:357] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.135625 22461 raft_consensus.cc:383] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.135691 22324 tablet_service.cc:1467] Processing CreateTablet for tablet f3743f45c75f4cc682f50eebda212633 (DEFAULT_TABLE table=default.Table_e24a40491b8f455eb04b0a4e567df5d5 [id=2f602150372549818fb7e4787b8a823d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.145783 22458 tablet_bootstrap.cc:654] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.103428 23103 ts_tablet_manager.cc:1905] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.156437 22456 ts_tablet_manager.cc:1428] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20250112 08:57:12.136773 22324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3743f45c75f4cc682f50eebda212633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.135880 22461 raft_consensus.cc:738] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.157545 22461 consensus_queue.cc:260] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.138033 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a8864649341c889d805c5599c12af (DEFAULT_TABLE table=default.Table_03c9aad903a742bb96dd314764735ea0 [id=17cbe93491d6453682dc84a7e657bfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.139014 22337 tablet_service.cc:1467] Processing CreateTablet for tablet b7c3c7162ff144a79715944b7c9f0371 (DEFAULT_TABLE table=default.Table_e72a7add315b4e81a640003870fc274a [id=8f203a79876b4e0db644cd1c7792d0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.164855 22333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2a8864649341c889d805c5599c12af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.139990 22325 tablet_service.cc:1467] Processing CreateTablet for tablet edf7a555e58c4a58b936e7e03ef6d955 (DEFAULT_TABLE table=default.Table_f3352b4abff244e88e533d442b24437d [id=9e070fa9e42c47adb9f1ae8fe5e194e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.141163 22327 tablet_service.cc:1467] Processing CreateTablet for tablet 790ac01251084cca90a7db0be0367fa2 (DEFAULT_TABLE table=default.Table_8597641f7fe6456fb720caf6be070864 [id=170c9bdb5e2c4c3c996505274364cf99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.141703 22332 tablet_service.cc:1467] Processing CreateTablet for tablet 1868b5cbb9a24196a47d4e8e6066bf78 (DEFAULT_TABLE table=default.Table_9bf70ac0df104d21b87badcbe3f09bf8 [id=f4382d0106df4ad49859bf940699099c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:12.169564 21702 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d686b08555414954aba52ea077889660 consensus state has no leader
I20250112 08:57:12.141795 23209 consensus_queue.cc:1035] T 53167910ab0a4accb1d1615d537ba7ae P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20250112 08:57:12.175212 22337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c3c7162ff144a79715944b7c9f0371. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.188385 22327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790ac01251084cca90a7db0be0367fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.131760 22565 ts_tablet_manager.cc:1939] T 267ab3e5955b4d3bbeca44baf021b328 P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
I20250112 08:57:12.197791 22332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1868b5cbb9a24196a47d4e8e6066bf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.142233 22336 tablet_service.cc:1467] Processing CreateTablet for tablet 89eb27e51282465cb03f2f9125afcb5e (DEFAULT_TABLE table=default.Table_214464b532cf4fc997136cca2adb554a [id=1912a594d679412db79ac9996db3cd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.142840 22335 tablet_service.cc:1467] Processing CreateTablet for tablet 504b71c1cdd84c019774a9c9d8ab2b11 (DEFAULT_TABLE table=default.Table_e0510c7c9f7b4a39b4229444845528fd [id=355da5876a5640249a35d1b09fdbce11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:12.142057 22290 leader_election.cc:304] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
W20250112 08:57:12.202041 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 60053us
I20250112 08:57:12.202651 22290 leader_election.cc:304] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.210772 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 194) took 4712 ms. Trace:
I20250112 08:57:12.211107 22565 rpcz_store.cc:276] 0112 08:57:07.498432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.498566 (+ 134us) service_pool.cc:225] Handling call
0112 08:57:11.933421 (+4434855us) ts_tablet_manager.cc:1145] Deleting tablet 267ab3e5955b4d3bbeca44baf021b328
0112 08:57:11.933454 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.056980 (+123526us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.130555 (+ 73575us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.210713 (+ 80158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12410,"tablet-delete.queue_time_us":4434278}
I20250112 08:57:12.212319 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 267ab3e5955b4d3bbeca44baf021b328 (table default.Table_88d7d1b5011d47be821f6e0e76f02d3e [id=bda1c38dfb7040ea8876b258b5ccf6b2]) successfully deleted
I20250112 08:57:12.213358 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 197) took 4445 ms. Trace:
I20250112 08:57:12.213654 22565 rpcz_store.cc:276] 0112 08:57:07.768041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:07.768211 (+ 170us) service_pool.cc:225] Handling call
0112 08:57:12.213031 (+4444820us) ts_tablet_manager.cc:1145] Deleting tablet 5cd402674080464cb3bde3697ac23180
0112 08:57:12.213073 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.213309 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4444156}
W20250112 08:57:12.214737 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 5cd402674080464cb3bde3697ac23180 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cd402674080464cb3bde3697ac23180 already in progress: creating tablet
I20250112 08:57:12.226159 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ddac23210248e3be1aba6591641b2d" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:12.227062 22565 tablet_replica.cc:331] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: stopping tablet replica
I20250112 08:57:12.233084 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c228f04faa5c4f68a95adfb295235047" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.241883 21999 leader_election.cc:304] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.243330 23209 raft_consensus.cc:2743] T 7d93c5e0d2a14c16a8080b21da64eb85 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.244405 23188 raft_consensus.cc:2798] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:12.244822 23188 raft_consensus.cc:491] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:12.245160 23188 raft_consensus.cc:3054] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.249379 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340f93f630a34b3b9a9386999e2684e8" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.248466 23283 raft_consensus.cc:3054] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.253610 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.255048 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74437c3dea6547f3b9545da3f8f971bb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.256469 22565 raft_consensus.cc:2238] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.256757 22456 tablet_bootstrap.cc:492] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:12.256942 22565 raft_consensus.cc:2267] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:12.258622 22565 ts_tablet_manager.cc:1905] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.259328 23281 consensus_queue.cc:1035] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:12.260975 22336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89eb27e51282465cb03f2f9125afcb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.262101 22456 tablet_bootstrap.cc:654] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:12.271067 22146 leader_election.cc:343] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.271548 22146 leader_election.cc:304] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.272329 23264 raft_consensus.cc:2743] T c228f04faa5c4f68a95adfb295235047 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:12.273067 22146 leader_election.cc:343] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.273424 22146 leader_election.cc:304] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.273993 23264 raft_consensus.cc:2743] T 340f93f630a34b3b9a9386999e2684e8 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:12.274719 22146 leader_election.cc:343] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.277860 22458 tablet_bootstrap.cc:492] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:12.278443 22458 ts_tablet_manager.cc:1397] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.611s user 0.015s sys 0.006s
I20250112 08:57:12.146811 23268 raft_consensus.cc:491] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.280691 23268 raft_consensus.cc:513] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.280984 22458 raft_consensus.cc:357] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.281742 22458 raft_consensus.cc:383] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.282027 22458 raft_consensus.cc:738] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.282465 23268 leader_election.cc:290] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.282665 22458 consensus_queue.cc:260] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.284365 22461 ts_tablet_manager.cc:1428] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.152s user 0.003s sys 0.004s
I20250112 08:57:12.284958 22458 ts_tablet_manager.cc:1428] T ea15d9405b9944db8e00121965b9ffde P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:12.286027 22458 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:57:12.278765 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 75212.2us
I20250112 08:57:12.287263 23281 raft_consensus.cc:491] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.288924 22290 leader_election.cc:304] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.288759 23281 raft_consensus.cc:513] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.291163 23281 leader_election.cc:290] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.293185 22325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf7a555e58c4a58b936e7e03ef6d955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.300086 23268 raft_consensus.cc:2798] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:12.300531 23268 raft_consensus.cc:695] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:12.301317 23268 consensus_queue.cc:237] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.310905 23103 ts_tablet_manager.cc:1918] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:12.311295 23103 log.cc:1198] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/06717cb567a54ee6a268c3be5d8786ec
W20250112 08:57:12.318519 22146 leader_election.cc:343] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.325233 22456 tablet_bootstrap.cc:492] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:12.325368 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c6ea87589249739461e629eee5f40b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:12.325620 22456 ts_tablet_manager.cc:1397] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.001s
I20250112 08:57:12.325945 22355 raft_consensus.cc:2463] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 0.
I20250112 08:57:12.327574 22290 leader_election.cc:304] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:12.327965 22456 raft_consensus.cc:357] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.328266 22002 leader_election.cc:304] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:12.328613 22456 raft_consensus.cc:383] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.328845 22456 raft_consensus.cc:738] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.328938 23209 raft_consensus.cc:2798] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:12.329259 23209 raft_consensus.cc:491] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:12.329200 23096 consensus_queue.cc:1035] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:12.329576 23209 raft_consensus.cc:3054] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.329465 22456 consensus_queue.cc:260] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.330463 22458 tablet_bootstrap.cc:654] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.330433 23096 raft_consensus.cc:2798] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:12.330801 23096 raft_consensus.cc:491] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:12.331099 23096 raft_consensus.cc:3054] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.331701 22456 ts_tablet_manager.cc:1428] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:12.337078 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504b71c1cdd84c019774a9c9d8ab2b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:12.312044 23103 ts_tablet_manager.cc:1939] T 06717cb567a54ee6a268c3be5d8786ec P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:12.303795 23188 raft_consensus.cc:513] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.345121 23096 raft_consensus.cc:513] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.347621 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c6ea87589249739461e629eee5f40b" candidate_uuid: "76718d1596134c83900541c6d398d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:12.348188 22352 raft_consensus.cc:3054] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.336167 23209 raft_consensus.cc:513] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:12.353710 22290 leader_election.cc:343] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 1df6282224b849a488071323e2e69e1c
I20250112 08:57:12.357498 23283 raft_consensus.cc:2743] T 98ddac23210248e3be1aba6591641b2d P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.358733 22456 tablet_bootstrap.cc:492] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:12.354699 22352 raft_consensus.cc:2463] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76718d1596134c83900541c6d398d59e in term 1.
I20250112 08:57:12.362169 23264 raft_consensus.cc:491] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.362617 23264 raft_consensus.cc:513] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.364606 23264 leader_election.cc:290] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.368117 23096 leader_election.cc:290] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.355276 23209 leader_election.cc:290] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.356283 23188 leader_election.cc:290] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.375523 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49ec037c22a42108ecb48bc8270191f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.376571 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
W20250112 08:57:12.376747 22146 leader_election.cc:343] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.378078 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
W20250112 08:57:12.378691 21999 leader_election.cc:343] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 80c6ea87589249739461e629eee5f40b
I20250112 08:57:12.379393 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
W20250112 08:57:12.379662 21999 leader_election.cc:343] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 80c6ea87589249739461e629eee5f40b
I20250112 08:57:12.382324 21710 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 1c1c634513e14e818601894834c41378 (table default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a]). Expected version 4 got 0
I20250112 08:57:12.391049 22350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:12.391474 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74437c3dea6547f3b9545da3f8f971bb" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:12.393458 22002 leader_election.cc:304] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870
I20250112 08:57:12.393416 22351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49ec037c22a42108ecb48bc8270191f" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:12.394963 23096 raft_consensus.cc:2798] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:12.395630 23096 raft_consensus.cc:695] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [term 1 LEADER]: Becoming Leader. State: Replica: 76718d1596134c83900541c6d398d59e, State: Running, Role: LEADER
I20250112 08:57:12.396420 23096 consensus_queue.cc:237] T 80c6ea87589249739461e629eee5f40b P 76718d1596134c83900541c6d398d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.405282 22458 tablet_bootstrap.cc:492] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:12.405704 22461 tablet_bootstrap.cc:492] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:12.405695 22458 ts_tablet_manager.cc:1397] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.120s user 0.008s sys 0.005s
I20250112 08:57:12.406296 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 35f60d1f4bed4721b73deb9ba787417c (table default.Table_0432088806604c248cefd166a4922284 [id=2c30293285c34e3394c6b20bde6d6ae6]) successfully deleted
I20250112 08:57:12.408284 22458 raft_consensus.cc:357] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.412112 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 208) took 3175 ms. Trace:
I20250112 08:57:12.412463 23103 rpcz_store.cc:276] 0112 08:57:09.239804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.278158 (+ 38354us) service_pool.cc:225] Handling call
0112 08:57:12.084722 (+2806564us) ts_tablet_manager.cc:1145] Deleting tablet 06717cb567a54ee6a268c3be5d8786ec
0112 08:57:12.088285 (+ 3563us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.193387 (+105102us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.310677 (+117290us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.412050 (+101373us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66331,"tablet-delete.queue_time_us":2805997}
I20250112 08:57:12.408938 22458 raft_consensus.cc:383] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.414016 22458 raft_consensus.cc:738] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.414832 22458 consensus_queue.cc:260] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.420481 22458 ts_tablet_manager.cc:1428] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250112 08:57:12.421372 22458 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:12.427804 23103 tablet_replica.cc:331] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:12.428025 23264 raft_consensus.cc:491] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.428663 23103 raft_consensus.cc:2238] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.428462 23264 raft_consensus.cc:513] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.429150 23103 raft_consensus.cc:2267] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:12.431130 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aabb08ff90f42d186487dc5c99644d2" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.431411 23264 leader_election.cc:290] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:57:12.432421 22146 leader_election.cc:343] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.432917 22461 tablet_bootstrap.cc:654] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.435058 22458 tablet_bootstrap.cc:654] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.436111 23103 ts_tablet_manager.cc:1905] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.452961 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:12.455678 23283 raft_consensus.cc:491] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.456125 23283 raft_consensus.cc:513] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.457912 23283 leader_election.cc:290] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.458853 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d686b08555414954aba52ea077889660" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:12.459353 22461 tablet_bootstrap.cc:492] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:12.459744 22461 ts_tablet_manager.cc:1397] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.001s
I20250112 08:57:12.459707 22209 raft_consensus.cc:3054] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.461988 22461 raft_consensus.cc:357] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.462653 22461 raft_consensus.cc:383] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.462946 22461 raft_consensus.cc:738] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.465081 22565 ts_tablet_manager.cc:1918] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:12.465421 22565 log.cc:1198] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/422e1484cd84479abba824cc8a6fae1f
I20250112 08:57:12.466066 22565 ts_tablet_manager.cc:1939] T 422e1484cd84479abba824cc8a6fae1f P 76718d1596134c83900541c6d398d59e: Deleting consensus metadata
W20250112 08:57:12.468083 22002 leader_election.cc:343] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Not found: Tablet not found: 5cd402674080464cb3bde3697ac23180
I20250112 08:57:12.463567 22461 consensus_queue.cc:260] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.475103 22461 ts_tablet_manager.cc:1428] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250112 08:57:12.475564 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e"
I20250112 08:57:12.478538 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 204) took 4349 ms. Trace:
I20250112 08:57:12.478874 22565 rpcz_store.cc:276] 0112 08:57:08.118772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.121559 (+ 2787us) service_pool.cc:225] Handling call
0112 08:57:12.226914 (+4105355us) ts_tablet_manager.cc:1145] Deleting tablet 422e1484cd84479abba824cc8a6fae1f
0112 08:57:12.226956 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.361352 (+134396us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.464871 (+103519us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.467866 (+ 2995us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28626,"tablet-delete.queue_time_us":4104823}
I20250112 08:57:12.479835 23268 raft_consensus.cc:491] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.480144 21701 catalog_manager.cc:4872] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): tablet 422e1484cd84479abba824cc8a6fae1f (table default.Table_bcb5d680dc6047ae9d719a1428694045 [id=97f50a84b2bb4e30afd1ecffdf6784f2]) successfully deleted
I20250112 08:57:12.480304 23268 raft_consensus.cc:513] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.481532 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05401e13af414ce9bf8fdb6fce76e254" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:12.482074 22213 raft_consensus.cc:3054] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:12.482118 23268 leader_election.cc:290] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:12.484833 22458 tablet_bootstrap.cc:492] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:12.485224 22458 ts_tablet_manager.cc:1397] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.000s
I20250112 08:57:12.488515 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 202) took 4370 ms. Trace:
I20250112 08:57:12.488669 22213 raft_consensus.cc:2463] T 05401e13af414ce9bf8fdb6fce76e254 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:12.488832 22565 rpcz_store.cc:276] 0112 08:57:08.118091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.126244 (+ 8153us) service_pool.cc:225] Handling call
0112 08:57:12.488164 (+4361920us) ts_tablet_manager.cc:1145] Deleting tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9
0112 08:57:12.488204 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.488462 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4338401}
I20250112 08:57:12.487498 22458 raft_consensus.cc:357] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:12.489964 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbfa5ddd5a74cabaf8db1a423f4e7c9 already in progress: creating tablet
I20250112 08:57:12.490163 22458 raft_consensus.cc:383] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.490437 22458 raft_consensus.cc:738] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
W20250112 08:57:12.490527 22149 leader_election.cc:343] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.490921 22149 leader_election.cc:304] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.491892 23264 raft_consensus.cc:2743] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.492370 22458 consensus_queue.cc:260] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.493016 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 207) took 4195 ms. Trace:
I20250112 08:57:12.493335 22565 rpcz_store.cc:276] 0112 08:57:08.298111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.298225 (+ 114us) service_pool.cc:225] Handling call
0112 08:57:12.492705 (+4194480us) ts_tablet_manager.cc:1145] Deleting tablet c4fc4d2d0eaa47f6a02f518828253057
0112 08:57:12.492742 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.492964 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4176330}
W20250112 08:57:12.493721 22149 leader_election.cc:343] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.494042 22149 leader_election.cc:304] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:12.494395 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet c4fc4d2d0eaa47f6a02f518828253057 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4fc4d2d0eaa47f6a02f518828253057 already in progress: creating tablet
I20250112 08:57:12.494885 23264 raft_consensus.cc:2743] T 74437c3dea6547f3b9545da3f8f971bb P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.496572 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 209) took 3945 ms. Trace:
I20250112 08:57:12.496846 22565 rpcz_store.cc:276] 0112 08:57:08.551495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.551594 (+ 99us) service_pool.cc:225] Handling call
0112 08:57:12.496278 (+3944684us) ts_tablet_manager.cc:1145] Deleting tablet 1ad1f85596894fe9992ec4ad115e6928
0112 08:57:12.496321 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.496528 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3943885}
W20250112 08:57:12.497903 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 1ad1f85596894fe9992ec4ad115e6928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad1f85596894fe9992ec4ad115e6928 already in progress: creating tablet
I20250112 08:57:12.499280 22458 ts_tablet_manager.cc:1428] T 2e62435dce1441a3b2fee50518662d48 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250112 08:57:12.500090 22458 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
W20250112 08:57:12.501240 22149 leader_election.cc:343] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.501642 22149 leader_election.cc:304] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.502599 23264 raft_consensus.cc:2743] T c49ec037c22a42108ecb48bc8270191f P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.504418 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 212) took 2780 ms. Trace:
I20250112 08:57:12.504732 22565 rpcz_store.cc:276] 0112 08:57:09.723827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.723998 (+ 171us) service_pool.cc:225] Handling call
0112 08:57:12.504150 (+2780152us) ts_tablet_manager.cc:1145] Deleting tablet 65591a0bb44148c49715b4564dd13524
0112 08:57:12.504190 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.504364 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2779630}
I20250112 08:57:12.505209 22458 tablet_bootstrap.cc:654] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:12.505872 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 65591a0bb44148c49715b4564dd13524 because the tablet was not found. No further retry: Not found: Tablet not found: 65591a0bb44148c49715b4564dd13524
I20250112 08:57:12.506825 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 211) took 2783 ms. Trace:
I20250112 08:57:12.507098 22565 rpcz_store.cc:276] 0112 08:57:09.723576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.872040 (+148464us) service_pool.cc:225] Handling call
0112 08:57:12.506621 (+2634581us) ts_tablet_manager.cc:1145] Deleting tablet 06717cb567a54ee6a268c3be5d8786ec
0112 08:57:12.506655 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.506780 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618803}
W20250112 08:57:12.508170 21701 catalog_manager.cc:4839] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 06717cb567a54ee6a268c3be5d8786ec because the tablet was not found. No further retry: Not found: Tablet not found: 06717cb567a54ee6a268c3be5d8786ec
I20250112 08:57:12.509099 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 217) took 2435 ms. Trace:
I20250112 08:57:12.509397 22565 rpcz_store.cc:276] 0112 08:57:10.074185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:10.074298 (+ 113us) service_pool.cc:225] Handling call
0112 08:57:12.508776 (+2434478us) ts_tablet_manager.cc:1145] Deleting tablet 575891fa458d4d8391d493dbe34fc0b3
0112 08:57:12.508809 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.509038 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2433987}
W20250112 08:57:12.510394 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet 575891fa458d4d8391d493dbe34fc0b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575891fa458d4d8391d493dbe34fc0b3 already in progress: creating tablet
W20250112 08:57:12.511750 22290 leader_election.cc:343] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.514928 22565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56484 (request call id 233) took 1996 ms. Trace:
I20250112 08:57:12.515200 22565 rpcz_store.cc:276] 0112 08:57:10.518402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:10.518541 (+ 139us) service_pool.cc:225] Handling call
0112 08:57:12.514659 (+1996118us) ts_tablet_manager.cc:1145] Deleting tablet c49ec037c22a42108ecb48bc8270191f
0112 08:57:12.514695 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.514885 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1995205}
W20250112 08:57:12.516224 21701 catalog_manager.cc:4852] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): delete failed for tablet c49ec037c22a42108ecb48bc8270191f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49ec037c22a42108ecb48bc8270191f already in progress: creating tablet
I20250112 08:57:12.531221 22458 tablet_bootstrap.cc:492] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e: No bootstrap required, opened a new log
I20250112 08:57:12.531647 22458 ts_tablet_manager.cc:1397] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
W20250112 08:57:12.533892 21999 leader_election.cc:343] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 5cd402674080464cb3bde3697ac23180
I20250112 08:57:12.514313 22461 tablet_bootstrap.cc:492] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:12.539754 22290 leader_election.cc:343] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.546078 21999 leader_election.cc:304] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76718d1596134c83900541c6d398d59e; no voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:12.547784 21713 catalog_manager.cc:5526] T 7d93c5e0d2a14c16a8080b21da64eb85 P 76718d1596134c83900541c6d398d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 76718d1596134c83900541c6d398d59e (127.21.42.193). New cstate: current_term: 1 leader_uuid: "76718d1596134c83900541c6d398d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:12.550451 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:12.551600 23264 raft_consensus.cc:491] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.552120 23264 raft_consensus.cc:513] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.554252 23264 leader_election.cc:290] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.534080 22458 raft_consensus.cc:357] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.555979 22062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46707de727a49108afb12ea2cf08aa0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:12.569255 22002 consensus_peers.cc:487] T a44cad3bd5a84e75a41733d8d2f345a8 P 76718d1596134c83900541c6d398d59e -> Peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Couldn't send request to peer 8c608785d6874abf9cc28db11a979ddd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:12.577102 22146 leader_election.cc:343] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.584173 22458 raft_consensus.cc:383] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.584573 22458 raft_consensus.cc:738] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76718d1596134c83900541c6d398d59e, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.585434 22458 consensus_queue.cc:260] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.587612 22458 ts_tablet_manager.cc:1428] T 05401e13af414ce9bf8fdb6fce76e254 P 76718d1596134c83900541c6d398d59e: Time spent starting tablet: real 0.056s user 0.006s sys 0.001s
I20250112 08:57:12.544581 22461 tablet_bootstrap.cc:654] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.642829 22290 leader_election.cc:304] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.645066 23188 raft_consensus.cc:2743] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.619431 23268 raft_consensus.cc:491] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:57:12.643779 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 104062us
I20250112 08:57:12.646137 23268 raft_consensus.cc:513] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:57:12.646991 22290 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Not found: Tablet not found: 0b38deeff1964b09938a12cfd4992ec5
W20250112 08:57:12.647583 22290 leader_election.cc:343] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.647976 23268 leader_election.cc:290] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.657452 22458 tablet_bootstrap.cc:492] T ef9c33e005b645e29464178c92a2466a P 76718d1596134c83900541c6d398d59e: Bootstrap starting.
I20250112 08:57:12.665364 23307 tablet.cc:1721] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 08:57:12.671057 22461 tablet_bootstrap.cc:492] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:12.671589 22461 ts_tablet_manager.cc:1397] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.157s user 0.011s sys 0.005s
I20250112 08:57:12.674414 23264 raft_consensus.cc:491] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.674889 23264 raft_consensus.cc:513] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.674489 22461 raft_consensus.cc:357] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.676790 23264 leader_election.cc:290] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.676987 22461 raft_consensus.cc:383] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.677217 22461 raft_consensus.cc:738] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.684466 22461 consensus_queue.cc:260] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.706030 22461 ts_tablet_manager.cc:1428] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20250112 08:57:12.706969 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b9c882084142118fe251e6540fd8be" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:12.721204 22146 leader_election.cc:343] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.733732 22461 tablet_bootstrap.cc:492] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:12.743854 21999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.194:34135, user_credentials={real_user=slave}} blocked reactor thread for 209978us
I20250112 08:57:12.748351 22209 raft_consensus.cc:2463] T d686b08555414954aba52ea077889660 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
I20250112 08:57:12.749758 23280 raft_consensus.cc:2743] T 5cd402674080464cb3bde3697ac23180 P 76718d1596134c83900541c6d398d59e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.764104 23307 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:44720 (request call id 233) took 1298 ms. Trace:
I20250112 08:57:12.764482 23307 rpcz_store.cc:276] 0112 08:57:11.465322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:11.469727 (+ 4405us) service_pool.cc:225] Handling call
0112 08:57:12.764031 (+1294304us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:11.583693 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 08:57:11.584028 (+ 335us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 08:57:11.584055 (+ 27us) alter_schema_op.cc:66] Acquired schema lock
0112 08:57:11.584123 (+ 68us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 08:57:11.613338 (+ 29215us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b2c00005778. stack: 00007fe54c963853 00007fe54c963750 00007fe54c4d8584 00000000004fcd24 00000000004fccb9 00007fe552bea741 00007fe552bf940a 00007fe552bf9093 00007fe552bf9031 00007fe55391a042 00007fe553919e1d 00007fe553d46073 00007fe553d45b88 00007fe553d4a821 00007fe553d4a7d9 00007fe553d4a769
0112 08:57:11.613410 (+ 72us) alter_schema_op.cc:128] START. Timestamp: 1736672231588109
0112 08:57:11.613431 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 08:57:11.614014 (+ 583us) log.cc:844] Serialized 173 byte log entry
0112 08:57:12.649374 (+1035360us) op_driver.cc:464] REPLICATION: finished
0112 08:57:12.665246 (+ 15872us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 08:57:12.675987 (+ 10741us) log.cc:844] Serialized 5 byte log entry
0112 08:57:12.742256 (+ 66269us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.763134 (+ 20878us) log.cc:844] Serialized 14 byte log entry
0112 08:57:12.763885 (+ 751us) alter_schema_op.cc:73] Released schema lock
0112 08:57:12.763930 (+ 45us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15690,"prepare.queue_time_us":58541,"prepare.run_cpu_time_us":2793,"prepare.run_wall_time_us":30549,"replication_time_us":1035829,"spinlock_wait_cycles":14471936,"thread_start_us":166849,"threads_started":3,"wal-append.queue_time_us":36610}]]}
I20250112 08:57:12.767789 22461 tablet_bootstrap.cc:654] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.776629 23309 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:12.777079 23309 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:12.777940 23264 raft_consensus.cc:491] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.778371 23264 raft_consensus.cc:513] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.780200 23264 leader_election.cc:290] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.780743 21802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: false dest_uuid: "5fc0c669abed404c80d7c4cf4769f55a" is_pre_election: true
I20250112 08:57:12.782198 23309 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
I20250112 08:57:12.782560 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e8f51e50994929b40a5e0679b07d8c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
W20250112 08:57:12.784173 22146 leader_election.cc:343] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.780854 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: false dest_uuid: "15d38686b2c64b20ba87275e3ba176ab" is_pre_election: true
I20250112 08:57:12.786448 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:12.786962 21713 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:12.790225 21714 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: Sending DeleteTablet for 3 replicas of tablet e61e80e259db4645942141754c8491a3
I20250112 08:57:12.791594 21714 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56650 (request call id 288) took 4158 ms. Trace:
I20250112 08:57:12.791877 21714 rpcz_store.cc:276] 0112 08:57:08.633483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.760135 (+126652us) service_pool.cc:225] Handling call
0112 08:57:09.223027 (+462892us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 08:57:09.231017 (+ 7990us) catalog_manager.cc:2753] Modifying in-memory table state
0112 08:57:09.231411 (+ 394us) catalog_manager.cc:2762] Locking tablets
0112 08:57:10.003203 (+771792us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 08:57:10.268773 (+265570us) catalog_manager.cc:2794] Removing table from by-name map
0112 08:57:10.268987 (+ 214us) catalog_manager.cc:2810] Committing in-memory state
0112 08:57:12.790123 (+2521136us) catalog_manager.cc:2814] Aborting table tasks
0112 08:57:12.791540 (+ 1417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:10.022021 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:10.022344 (+ 323us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:10.022371 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:10.022383 (+ 12us) tablet.cc:659] Decoding operations
0112 08:57:10.022669 (+ 286us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:10.022697 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:10.022716 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0112 08:57:10.022977 (+ 261us) tablet.cc:698] Row locks acquired
0112 08:57:10.022990 (+ 13us) write_op.cc:260] PREPARE: finished
0112 08:57:10.023125 (+ 135us) write_op.cc:270] Start()
0112 08:57:10.023198 (+ 73us) write_op.cc:276] Timestamp: P: 1736672230023041 usec, L: 0
0112 08:57:10.023213 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 08:57:10.023618 (+ 405us) log.cc:844] Serialized 737 byte log entry
0112 08:57:10.259151 (+235533us) op_driver.cc:464] REPLICATION: finished
0112 08:57:10.262986 (+ 3835us) write_op.cc:301] APPLY: starting
0112 08:57:10.263050 (+ 64us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:10.263200 (+ 150us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:10.263306 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:10.263597 (+ 291us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:10.263766 (+ 169us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 08:57:10.263825 (+ 59us) write_op.cc:312] APPLY: finished
0112 08:57:10.264034 (+ 209us) log.cc:844] Serialized 29 byte log entry
0112 08:57:10.264181 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.264248 (+ 67us) write_op.cc:454] Released schema lock
0112 08:57:10.264287 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3676,"apply.run_cpu_time_us":1499,"apply.run_wall_time_us":1498,"num_ops":2,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":1763,"prepare.run_wall_time_us":1763,"replication_time_us":235852,"thread_start_us":17892,"threads_started":1}]]}
I20250112 08:57:12.797065 23264 raft_consensus.cc:491] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.797518 23264 raft_consensus.cc:513] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.800604 22067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1869844a4d2b4a019dadb26d5908b520" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76718d1596134c83900541c6d398d59e" is_pre_election: true
I20250112 08:57:12.801388 21675 master-stress-test.cc:579] Killed test thread 8/10
I20250112 08:57:12.801625 21675 master-stress-test.cc:577] Killing test thread 9/10
I20250112 08:57:12.801913 21675 master-stress-test.cc:579] Killed test thread 9/10
W20250112 08:57:12.801813 22146 leader_election.cc:343] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:12.802140 21675 master-stress-test.cc:577] Killing test thread 10/10
I20250112 08:57:12.802168 22036 tablet_service.cc:1514] Processing DeleteTablet for tablet e61e80e259db4645942141754c8491a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:56484
I20250112 08:57:12.802414 21675 master-stress-test.cc:579] Killed test thread 10/10
I20250112 08:57:12.802429 23264 leader_election.cc:290] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.802783 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 21954
I20250112 08:57:12.806735 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet e61e80e259db4645942141754c8491a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:12.817301 21715 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 47) took 4353 ms (client timeout 1000 ms). Trace:
W20250112 08:57:12.648177 22290 leader_election.cc:343] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Illegal state: must be running to vote when last-logged opid is not known
W20250112 08:57:12.820302 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 172134us
I20250112 08:57:12.822265 21708 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_76c1e51c857948f0b9bf826ca8c4dcd5 [id=1460b936c99743a3aca81d2d2a377fbb] alter complete (version 1)
I20250112 08:57:12.786818 22818 leader_election.cc:304] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: caa33dde8b5248e7ab4d6087d8e9d801; no voters: 15d38686b2c64b20ba87275e3ba176ab, 5fc0c669abed404c80d7c4cf4769f55a
W20250112 08:57:12.817643 21715 rpcz_store.cc:269] 0112 08:57:08.477514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.477637 (+ 123us) service_pool.cc:225] Handling call
0112 08:57:12.817231 (+4339594us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:10.694747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:10.695264 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:10.695294 (+ 30us) write_op.cc:435] Acquired schema lock
0112 08:57:10.695313 (+ 19us) tablet.cc:659] Decoding operations
0112 08:57:10.695863 (+ 550us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:10.695941 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:10.695971 (+ 30us) tablet.cc:682] Acquiring locks for 3 operations
0112 08:57:10.696304 (+ 333us) tablet.cc:698] Row locks acquired
0112 08:57:10.696323 (+ 19us) write_op.cc:260] PREPARE: finished
0112 08:57:10.696444 (+ 121us) write_op.cc:270] Start()
0112 08:57:10.696538 (+ 94us) write_op.cc:276] Timestamp: P: 1736672230696402 usec, L: 0
0112 08:57:10.696556 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 08:57:10.697135 (+ 579us) log.cc:844] Serialized 1293 byte log entry
0112 08:57:10.755985 (+ 58850us) op_driver.cc:464] REPLICATION: finished
0112 08:57:10.756383 (+ 398us) write_op.cc:301] APPLY: starting
0112 08:57:10.756443 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:10.756565 (+ 122us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:10.756676 (+ 111us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:10.757044 (+ 368us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:10.757198 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 08:57:10.757259 (+ 61us) write_op.cc:312] APPLY: finished
0112 08:57:10.757508 (+ 249us) log.cc:844] Serialized 35 byte log entry
0112 08:57:10.757651 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.757727 (+ 76us) write_op.cc:454] Released schema lock
0112 08:57:10.757757 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:10.757971 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.757993 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":275167,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":228,"apply.run_cpu_time_us":1580,"apply.run_wall_time_us":1580,"num_ops":3,"prepare.queue_time_us":36544,"prepare.run_cpu_time_us":2803,"prepare.run_wall_time_us":35154,"raft.queue_time_us":111,"raft.run_cpu_time_us":736,"raft.run_wall_time_us":736,"replication_time_us":59319,"thread_start_us":78338,"threads_started":1}]]}
I20250112 08:57:12.832139 23312 tablet_replica.cc:331] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:12.833029 23312 raft_consensus.cc:2238] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.833503 23312 raft_consensus.cc:2267] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:12.833698 23309 raft_consensus.cc:2743] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:12.834350 22181 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:12.834940 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:12.835232 23312 ts_tablet_manager.cc:1905] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.835515 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:12.822410 21711 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_275569887e7c49dd9608ddf9efc7fb39 [id=8172f1c5f3b54191aaa2322ccd8c638d] alter complete (version 1)
I20250112 08:57:12.836249 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:12.836702 22186 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:12.836689 21712 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 45) took 4373 ms (client timeout 1000 ms). Trace:
W20250112 08:57:12.837018 21712 rpcz_store.cc:269] 0112 08:57:08.463313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:08.463437 (+ 124us) service_pool.cc:225] Handling call
0112 08:57:12.836589 (+4373152us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":13}
W20250112 08:57:12.837668 22818 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.254:45457, user_credentials={real_user=slave}} blocked reactor thread for 50836.9us
I20250112 08:57:12.860273 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 06717cb567a54ee6a268c3be5d8786ec (table default.Table_c18e0bb841d942978709c4d445929483 [id=4e7fa7e2c76f4932bd3ae514033670e1]) successfully deleted
I20250112 08:57:12.868211 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46707de727a49108afb12ea2cf08aa0" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:12.868885 22209 raft_consensus.cc:2463] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
I20250112 08:57:12.872709 23312 ts_tablet_manager.cc:1918] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:12.873111 23312 log.cc:1198] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/e61e80e259db4645942141754c8491a3
I20250112 08:57:12.873790 23312 ts_tablet_manager.cc:1939] T e61e80e259db4645942141754c8491a3 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:12.875833 22209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870" is_pre_election: true
I20250112 08:57:12.876434 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet e61e80e259db4645942141754c8491a3 (table default.Table_a6bd091b79fd4a2c952ed2f0f95da386 [id=0b9a2155e0614d7b88621865898e5eb6]) successfully deleted
I20250112 08:57:12.876482 22209 raft_consensus.cc:2463] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 0.
W20250112 08:57:12.877871 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 422e1484cd84479abba824cc8a6fae1f because the tablet was not found. No further retry: Not found: Tablet not found: 422e1484cd84479abba824cc8a6fae1f
I20250112 08:57:12.878859 23312 tablet_replica.cc:331] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:12.884577 23312 raft_consensus.cc:2238] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.885236 23312 raft_consensus.cc:2267] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:12.886971 23312 ts_tablet_manager.cc:1905] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.891549 22461 tablet_bootstrap.cc:492] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:12.892038 22461 ts_tablet_manager.cc:1397] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.158s user 0.010s sys 0.000s
I20250112 08:57:12.892114 23103 ts_tablet_manager.cc:1918] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:12.892495 23103 log.cc:1198] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:12.893172 23103 ts_tablet_manager.cc:1939] T a44cad3bd5a84e75a41733d8d2f345a8 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:12.895401 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 210) took 3530 ms. Trace:
I20250112 08:57:12.905993 22461 raft_consensus.cc:357] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.912029 22461 raft_consensus.cc:383] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:12.912380 22461 raft_consensus.cc:738] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:12.913014 22461 consensus_queue.cc:260] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.908541 22325 tablet_service.cc:1514] Processing DeleteTablet for tablet e61e80e259db4645942141754c8491a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.914611 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.915486 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.915769 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f60d1f4bed4721b73deb9ba787417c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:56:59 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.924695 22461 ts_tablet_manager.cc:1428] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20250112 08:57:12.936619 22461 tablet_bootstrap.cc:492] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:12.936904 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b38deeff1964b09938a12cfd4992ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:03 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.937862 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet ea15d9405b9944db8e00121965b9ffde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.895812 23103 rpcz_store.cc:276] 0112 08:57:09.364729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.367334 (+ 2605us) service_pool.cc:225] Handling call
0112 08:57:12.413565 (+3046231us) ts_tablet_manager.cc:1145] Deleting tablet a44cad3bd5a84e75a41733d8d2f345a8
0112 08:57:12.418273 (+ 4708us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.744961 (+326688us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.891819 (+146858us) tablet_metadata.cc:641] Metadata flushed
0112 08:57:12.895321 (+ 3502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3042196}
I20250112 08:57:12.956553 22461 tablet_bootstrap.cc:654] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.958026 22456 tablet_bootstrap.cc:654] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:12.958047 23312 ts_tablet_manager.cc:1918] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:12.958611 23312 log.cc:1198] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/fc82026fcd1541418176a015ea3253cf
I20250112 08:57:12.959369 23312 ts_tablet_manager.cc:1939] T fc82026fcd1541418176a015ea3253cf P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:12.961994 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet fc82026fcd1541418176a015ea3253cf (table default.Table_dba2962989f24aec9af818bd34ea9122 [id=131b023798eb4e77a12306fb24329147]) successfully deleted
I20250112 08:57:12.963052 23312 tablet_replica.cc:331] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:12.964110 23312 raft_consensus.cc:2238] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:12.964635 23312 raft_consensus.cc:2267] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:12.964855 22290 leader_election.cc:343] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:12.965216 22290 leader_election.cc:304] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.966102 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd402674080464cb3bde3697ac23180 with delete_type TABLET_DATA_DELETED (replaced by tablet ae66efbd303d4a25b4c547915109bca7) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:12.967473 23312 ts_tablet_manager.cc:1905] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:12.972756 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 209) took 3679 ms. Trace:
I20250112 08:57:12.973090 23103 rpcz_store.cc:276] 0112 08:57:09.293128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.328493 (+ 35365us) service_pool.cc:225] Handling call
0112 08:57:12.972476 (+3643983us) ts_tablet_manager.cc:1145] Deleting tablet 267ab3e5955b4d3bbeca44baf021b328
0112 08:57:12.972530 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.972701 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590315}
I20250112 08:57:12.974397 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 214) took 1901 ms. Trace:
I20250112 08:57:12.974815 23103 rpcz_store.cc:276] 0112 08:57:11.072553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:11.072694 (+ 141us) service_pool.cc:225] Handling call
0112 08:57:12.974074 (+1901380us) ts_tablet_manager.cc:1145] Deleting tablet 575891fa458d4d8391d493dbe34fc0b3
0112 08:57:12.974110 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.974328 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900697}
I20250112 08:57:12.975968 23103 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44720 (request call id 229) took 1511 ms. Trace:
I20250112 08:57:12.976253 23103 rpcz_store.cc:276] 0112 08:57:11.464194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:11.507223 (+ 43029us) service_pool.cc:225] Handling call
0112 08:57:12.975673 (+1468450us) ts_tablet_manager.cc:1145] Deleting tablet c49ec037c22a42108ecb48bc8270191f
0112 08:57:12.975704 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 08:57:12.975923 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392332}
I20250112 08:57:12.977931 23290 raft_consensus.cc:2743] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:12.979053 22290 leader_election.cc:304] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.979740 23290 raft_consensus.cc:2798] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:12.970116 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:12.980218 23290 raft_consensus.cc:695] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:12.980978 23290 consensus_queue.cc:237] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:12.983323 22456 tablet_bootstrap.cc:492] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:12.986056 23264 raft_consensus.cc:491] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:12.988484 23264 raft_consensus.cc:513] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.990823 23264 leader_election.cc:290] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:12.992534 22456 ts_tablet_manager.cc:1397] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.634s user 0.015s sys 0.001s
I20250112 08:57:12.995019 22456 raft_consensus.cc:357] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:12.981490 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:12.996762 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 (table default.Table_c7e1f3d4703d41ffb340c43933e0fd41 [id=d3dfc1c9cf0b4d52a2e152226ce5273e]). Expected version 1 got 0
I20250112 08:57:12.997679 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 2 got 0
I20250112 08:57:12.998222 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c228f04faa5c4f68a95adfb295235047 (table default.Table_b576158684104afd8673ccbe72b45c40 [id=929d59bdba3141f5947bc3af54185497]). Expected version 1 got 0
I20250112 08:57:12.999114 22290 leader_election.cc:304] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:12.999699 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:12.999775 23290 raft_consensus.cc:2798] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:13.000494 23290 raft_consensus.cc:695] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:12.995666 22456 raft_consensus.cc:383] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.001543 22456 raft_consensus.cc:738] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.001267 23290 consensus_queue.cc:237] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.002256 22456 consensus_queue.cc:260] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.017812 23312 ts_tablet_manager.cc:1918] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.021973 22290 leader_election.cc:304] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.022680 23312 log.cc:1198] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/f46707de727a49108afb12ea2cf08aa0
I20250112 08:57:13.022724 23268 raft_consensus.cc:2798] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:13.023061 23268 raft_consensus.cc:491] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:13.023420 23268 raft_consensus.cc:3054] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.023614 23312 ts_tablet_manager.cc:1939] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.027974 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet f46707de727a49108afb12ea2cf08aa0 (table default.Table_397822bc7cf943e7a8ab1fcd91cd6a31 [id=644fe986709840f6aea68e8586428ba5]) successfully deleted
I20250112 08:57:13.028832 22456 ts_tablet_manager.cc:1428] T c4fc4d2d0eaa47f6a02f518828253057 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250112 08:57:13.029709 22456 tablet_bootstrap.cc:492] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:13.029953 23268 raft_consensus.cc:513] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.031783 23268 leader_election.cc:290] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:13.034157 22461 tablet_bootstrap.cc:492] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.034536 22461 ts_tablet_manager.cc:1397] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.002s
I20250112 08:57:13.034858 23312 tablet_replica.cc:331] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.035115 22456 tablet_bootstrap.cc:654] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.035887 23312 raft_consensus.cc:2238] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.036190 22290 leader_election.cc:304] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.037451 22351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aabb08ff90f42d186487dc5c99644d2" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.037662 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46707de727a49108afb12ea2cf08aa0" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.038393 22355 raft_consensus.cc:2388] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53068c99960647728141706e9c50b870 in current term 1: Already voted for candidate 8c608785d6874abf9cc28db11a979ddd in this term.
I20250112 08:57:13.038945 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 74437c3dea6547f3b9545da3f8f971bb (table default.Table_d1591f699c154e6f9124d57a6779e410 [id=3d1f3e4fa4004b17b07b0978a27aff3f]). Expected version 1 got 0
I20250112 08:57:13.040197 21714 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 340f93f630a34b3b9a9386999e2684e8 (table default.Table_1972c3e02af94e419ca99e14b22a365a [id=453899f3df2e41849bdc3cdd8c32b93b]). Expected version 1 got 0
I20250112 08:57:13.040549 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b9c882084142118fe251e6540fd8be" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.044732 23312 raft_consensus.cc:2267] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.050199 23268 raft_consensus.cc:2798] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:13.050586 23268 raft_consensus.cc:491] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:13.050904 23268 raft_consensus.cc:3054] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.053319 23312 ts_tablet_manager.cc:1905] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.055657 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:13.055760 22461 raft_consensus.cc:357] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.056432 22461 raft_consensus.cc:383] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.056691 22461 raft_consensus.cc:738] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.057264 22461 consensus_queue.cc:260] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.059937 22461 ts_tablet_manager.cc:1428] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250112 08:57:13.060806 22461 tablet_bootstrap.cc:492] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:13.064675 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e8f51e50994929b40a5e0679b07d8c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.065258 22355 raft_consensus.cc:2463] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:13.066092 22352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1869844a4d2b4a019dadb26d5908b520" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.066794 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c34fc4b6aaf45fdae76cf153129f2da" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.067371 22357 raft_consensus.cc:2463] T 6c34fc4b6aaf45fdae76cf153129f2da P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
W20250112 08:57:13.070312 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 267ab3e5955b4d3bbeca44baf021b328 because the tablet was not found. No further retry: Not found: Tablet not found: 267ab3e5955b4d3bbeca44baf021b328
W20250112 08:57:13.071245 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 575891fa458d4d8391d493dbe34fc0b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575891fa458d4d8391d493dbe34fc0b3 already in progress: creating tablet
W20250112 08:57:13.072203 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet c49ec037c22a42108ecb48bc8270191f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49ec037c22a42108ecb48bc8270191f already in progress: creating tablet
W20250112 08:57:13.073009 21704 catalog_manager.cc:4852] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet e61e80e259db4645942141754c8491a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e61e80e259db4645942141754c8491a3 already in progress: creating tablet
I20250112 08:57:13.076543 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 2 got 0
I20250112 08:57:13.080082 22461 tablet_bootstrap.cc:654] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.080592 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.081852 21715 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:13.083662 23312 ts_tablet_manager.cc:1918] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.084064 23312 log.cc:1198] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/6c34fc4b6aaf45fdae76cf153129f2da
I20250112 08:57:13.084714 23312 ts_tablet_manager.cc:1939] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.089900 23312 tablet_replica.cc:331] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.090878 23312 raft_consensus.cc:2238] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.091380 23312 raft_consensus.cc:2267] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:13.091742 22149 leader_election.cc:343] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:13.092202 22149 leader_election.cc:304] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:13.092911 23264 raft_consensus.cc:2743] T 7aabb08ff90f42d186487dc5c99644d2 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:13.093218 23312 ts_tablet_manager.cc:1905] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.093621 22149 leader_election.cc:304] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
W20250112 08:57:13.094172 22149 raft_consensus.cc:2681] T f46707de727a49108afb12ea2cf08aa0 P 53068c99960647728141706e9c50b870: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 08:57:13.096818 22149 leader_election.cc:343] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:13.097287 22149 leader_election.cc:304] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:13.098066 23264 raft_consensus.cc:2743] T 56b9c882084142118fe251e6540fd8be P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:13.086383 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 6c34fc4b6aaf45fdae76cf153129f2da (table default.Table_83fb15dd26114a99b446cc4d91c8dd14 [id=0996fef5d9ad4192ace2ea4ae5d04450]) successfully deleted
I20250112 08:57:13.099370 22149 leader_election.cc:304] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters: 76718d1596134c83900541c6d398d59e
I20250112 08:57:13.100207 23264 raft_consensus.cc:2798] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:13.100523 23264 raft_consensus.cc:491] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:13.100812 23264 raft_consensus.cc:3054] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:13.101517 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 1514074a19694aa0a42b968a29904c30 because the tablet was not found. No further retry: Not found: Tablet not found: 1514074a19694aa0a42b968a29904c30
W20250112 08:57:13.106177 22290 leader_election.cc:343] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: f46707de727a49108afb12ea2cf08aa0
I20250112 08:57:13.107580 23264 raft_consensus.cc:513] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.109930 23264 leader_election.cc:290] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:13.116907 23268 raft_consensus.cc:513] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.119127 23268 leader_election.cc:290] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Requested vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:57:13.120412 22149 leader_election.cc:343] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:13.120836 22149 leader_election.cc:304] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53068c99960647728141706e9c50b870; no voters: 76718d1596134c83900541c6d398d59e, 8c608785d6874abf9cc28db11a979ddd
I20250112 08:57:13.121822 22149 leader_election.cc:304] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
W20250112 08:57:13.122380 22149 raft_consensus.cc:2681] T 6c34fc4b6aaf45fdae76cf153129f2da P 53068c99960647728141706e9c50b870: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:57:13.123082 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.126181 23103 tablet_replica.cc:331] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.126680 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:13.126891 23103 raft_consensus.cc:2238] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.127267 21712 catalog_manager.cc:5618] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:13.127564 23103 raft_consensus.cc:2267] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.128895 21708 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_4cbe685a04de4b6f8f391e827247cfee [id=6fa448eed39749ffa607e4ca225e4fbe] alter complete (version 1)
I20250112 08:57:13.129241 23103 ts_tablet_manager.cc:1905] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.129272 23334 raft_consensus.cc:2743] T 1869844a4d2b4a019dadb26d5908b520 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:13.110522 23335 consensus_queue.cc:1035] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:13.132354 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 422e1484cd84479abba824cc8a6fae1f because the tablet was not found. No further retry: Not found: Tablet not found: 422e1484cd84479abba824cc8a6fae1f
I20250112 08:57:13.133828 22461 tablet_bootstrap.cc:492] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.134197 22461 ts_tablet_manager.cc:1397] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.006s
I20250112 08:57:13.139209 21713 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_fea71badc3904a0ea177f78ebd702d15 [id=711de3e68bc84e8687f455ec32d8259a] alter complete (version 4)
I20250112 08:57:13.142205 21716 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_eb6046dddf9e4a28a8254e74a7cce16a [id=78a10e1e86624d089f2b6866a4a2da03] alter complete (version 2)
I20250112 08:57:13.144045 23290 raft_consensus.cc:491] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:13.144486 23290 raft_consensus.cc:513] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.146250 23290 leader_election.cc:290] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53068c99960647728141706e9c50b870 (127.21.42.194:34135), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
W20250112 08:57:13.148756 21716 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 46) took 3685 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.151672 21709 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 48) took 3315 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.152105 21709 rpcz_store.cc:269] 0112 08:57:09.836666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.836808 (+ 142us) service_pool.cc:225] Handling call
0112 08:57:13.151613 (+3314805us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:11.004923 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:11.005382 (+ 459us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:11.005409 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:11.005423 (+ 14us) tablet.cc:659] Decoding operations
0112 08:57:11.005721 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:11.005780 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:11.005805 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:11.005960 (+ 155us) tablet.cc:698] Row locks acquired
0112 08:57:11.005977 (+ 17us) write_op.cc:260] PREPARE: finished
0112 08:57:11.006106 (+ 129us) write_op.cc:270] Start()
0112 08:57:11.006205 (+ 99us) write_op.cc:276] Timestamp: P: 1736672231006040 usec, L: 0
0112 08:57:11.006221 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 08:57:11.006770 (+ 549us) log.cc:844] Serialized 533 byte log entry
0112 08:57:11.028156 (+ 21386us) op_driver.cc:464] REPLICATION: finished
0112 08:57:11.028471 (+ 315us) write_op.cc:301] APPLY: starting
0112 08:57:11.028539 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:11.028626 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:11.028718 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:11.028899 (+ 181us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:11.029030 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:11.029069 (+ 39us) write_op.cc:312] APPLY: finished
0112 08:57:11.029284 (+ 215us) log.cc:844] Serialized 23 byte log entry
0112 08:57:11.029426 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:11.029493 (+ 67us) write_op.cc:454] Released schema lock
0112 08:57:11.029523 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:11.029729 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:11.029753 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":44168,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":179,"apply.run_cpu_time_us":1230,"apply.run_wall_time_us":1230,"mutex_wait_us":7665,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2193,"prepare.run_wall_time_us":9839,"raft.queue_time_us":7887,"raft.run_cpu_time_us":632,"raft.run_wall_time_us":632,"replication_time_us":21835}]]}
I20250112 08:57:13.149622 22461 raft_consensus.cc:357] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.160786 22186 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.161435 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.161993 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.162663 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e924f523524bd0a387597399e8470a with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.163378 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.164260 22461 raft_consensus.cc:383] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.164548 22461 raft_consensus.cc:738] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
W20250112 08:57:13.166252 21717 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 47) took 2319 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.166565 21717 rpcz_store.cc:269] 0112 08:57:10.847050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:10.960827 (+113777us) service_pool.cc:225] Handling call
0112 08:57:13.166185 (+2205358us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":27}
I20250112 08:57:13.168232 22461 consensus_queue.cc:260] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.169320 22212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9c33e005b645e29464178c92a2466a" candidate_uuid: "8c608785d6874abf9cc28db11a979ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53068c99960647728141706e9c50b870"
I20250112 08:57:13.169849 22212 raft_consensus.cc:3054] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.170264 22186 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b34b09f56d4417a3f321599170e5fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.170400 22461 ts_tablet_manager.cc:1428] T f9835288a7974e85821b095c03e91886 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.036s user 0.004s sys 0.002s
I20250112 08:57:13.170850 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet f9835288a7974e85821b095c03e91886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.171216 22461 tablet_bootstrap.cc:492] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:13.171830 21711 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 44) took 3512 ms (client timeout 1000 ms). Trace:
I20250112 08:57:13.172073 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet e61e80e259db4645942141754c8491a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:13.172159 21711 rpcz_store.cc:269] 0112 08:57:09.659143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.659281 (+ 138us) service_pool.cc:225] Handling call
0112 08:57:10.957840 (+1298559us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.957867 (+ 27us) write_op.cc:454] Released schema lock
0112 08:57:13.171760 (+2213893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:10.883619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:10.884074 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:10.884102 (+ 28us) write_op.cc:435] Acquired schema lock
0112 08:57:10.884118 (+ 16us) tablet.cc:659] Decoding operations
0112 08:57:10.884417 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:10.884459 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:10.884482 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:10.884627 (+ 145us) tablet.cc:698] Row locks acquired
0112 08:57:10.884645 (+ 18us) write_op.cc:260] PREPARE: finished
0112 08:57:10.884786 (+ 141us) write_op.cc:270] Start()
0112 08:57:10.884871 (+ 85us) write_op.cc:276] Timestamp: P: 1736672230884727 usec, L: 0
0112 08:57:10.884889 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 08:57:10.885413 (+ 524us) log.cc:844] Serialized 356 byte log entry
0112 08:57:10.911846 (+ 26433us) op_driver.cc:464] REPLICATION: finished
0112 08:57:10.912182 (+ 336us) write_op.cc:301] APPLY: starting
0112 08:57:10.912233 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:10.912316 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:10.912411 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:10.912610 (+ 199us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:10.912730 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:10.912771 (+ 41us) write_op.cc:312] APPLY: finished
0112 08:57:10.912937 (+ 166us) log.cc:844] Serialized 23 byte log entry
0112 08:57:10.913083 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:10.913141 (+ 58us) write_op.cc:454] Released schema lock
0112 08:57:10.913175 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":26005,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":200,"apply.run_cpu_time_us":1152,"apply.run_wall_time_us":1152,"num_ops":1,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":2055,"prepare.run_wall_time_us":2062,"raft.queue_time_us":7395,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":26867}]]}
W20250112 08:57:13.182605 21713 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 46) took 1502 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.185343 21708 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.193:36265 (request call id 45) took 2414 ms (client timeout 1000 ms). Trace:
I20250112 08:57:13.189550 22212 raft_consensus.cc:2463] T ef9c33e005b645e29464178c92a2466a P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c608785d6874abf9cc28db11a979ddd in term 1.
W20250112 08:57:13.149806 21716 rpcz_store.cc:269] 0112 08:57:09.463539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:09.463694 (+ 155us) service_pool.cc:225] Handling call
0112 08:57:09.553274 (+ 89580us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26572dee 00007fcf242d0fc4 00007fcf242d0f52 00007fcf242d0e81 00007fcf242d0dfc 00007fcf242d00b2 00007fcf2259adfc 00007fcf2259a236
0112 08:57:13.148696 (+3595422us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:12.876772 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:12.877230 (+ 458us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:12.877255 (+ 25us) write_op.cc:435] Acquired schema lock
0112 08:57:12.877272 (+ 17us) tablet.cc:659] Decoding operations
0112 08:57:12.877692 (+ 420us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:12.877744 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:12.877772 (+ 28us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:12.877934 (+ 162us) tablet.cc:698] Row locks acquired
0112 08:57:12.877950 (+ 16us) write_op.cc:260] PREPARE: finished
0112 08:57:12.878085 (+ 135us) write_op.cc:270] Start()
0112 08:57:12.878173 (+ 88us) write_op.cc:276] Timestamp: P: 1736672232877988 usec, L: 0
0112 08:57:12.878194 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 08:57:12.878704 (+ 510us) log.cc:844] Serialized 356 byte log entry
0112 08:57:12.945307 (+ 66603us) op_driver.cc:464] REPLICATION: finished
0112 08:57:12.971993 (+ 26686us) write_op.cc:301] APPLY: starting
0112 08:57:12.972052 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:12.972170 (+ 118us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:12.972261 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:12.972411 (+ 150us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:12.972532 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:12.972570 (+ 38us) write_op.cc:312] APPLY: finished
0112 08:57:12.972729 (+ 159us) log.cc:844] Serialized 23 byte log entry
0112 08:57:12.972873 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.972923 (+ 50us) write_op.cc:454] Released schema lock
0112 08:57:12.972950 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":21,"spinlock_wait_cycles":169932160,"child_traces":[["op",{"apply.queue_time_us":26548,"apply.run_cpu_time_us":1144,"apply.run_wall_time_us":1144,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2151,"prepare.run_wall_time_us":2152,"replication_time_us":67036,"thread_start_us":53820,"threads_started":1}]]}
I20250112 08:57:13.193107 23331 tablet.cc:1721] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 53068c99960647728141706e9c50b870: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 08:57:13.194675 21710 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 49) took 1871 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.194990 21710 rpcz_store.cc:269] 0112 08:57:11.323447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:11.323565 (+ 118us) service_pool.cc:225] Handling call
0112 08:57:13.194613 (+1871048us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33065,"reported_tablets":28}
W20250112 08:57:13.185652 21708 rpcz_store.cc:269] 0112 08:57:10.771151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:10.773827 (+ 2676us) service_pool.cc:225] Handling call
0112 08:57:12.203812 (+1429985us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.203837 (+ 25us) write_op.cc:454] Released schema lock
0112 08:57:13.185260 (+981423us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:12.158264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:12.158700 (+ 436us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:12.158727 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:12.158743 (+ 16us) tablet.cc:659] Decoding operations
0112 08:57:12.159043 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:12.159085 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:12.159114 (+ 29us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:12.159272 (+ 158us) tablet.cc:698] Row locks acquired
0112 08:57:12.159288 (+ 16us) write_op.cc:260] PREPARE: finished
0112 08:57:12.159424 (+ 136us) write_op.cc:270] Start()
0112 08:57:12.159512 (+ 88us) write_op.cc:276] Timestamp: P: 1736672232159348 usec, L: 0
0112 08:57:12.159531 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 08:57:12.160084 (+ 553us) log.cc:844] Serialized 356 byte log entry
0112 08:57:12.173264 (+ 13180us) op_driver.cc:464] REPLICATION: finished
0112 08:57:12.173585 (+ 321us) write_op.cc:301] APPLY: starting
0112 08:57:12.173636 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:12.173731 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:12.173815 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:12.173994 (+ 179us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:12.174119 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:12.174164 (+ 45us) write_op.cc:312] APPLY: finished
0112 08:57:12.174342 (+ 178us) log.cc:844] Serialized 23 byte log entry
0112 08:57:12.174476 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.174533 (+ 57us) write_op.cc:454] Released schema lock
0112 08:57:12.174569 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 08:57:12.860463 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:12.861015 (+ 552us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:12.861043 (+ 28us) write_op.cc:435] Acquired schema lock
0112 08:57:12.861146 (+ 103us) tablet.cc:659] Decoding operations
0112 08:57:12.861443 (+ 297us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:12.861486 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:12.861510 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:12.861656 (+ 146us) tablet.cc:698] Row locks acquired
0112 08:57:12.861672 (+ 16us) write_op.cc:260] PREPARE: finished
0112 08:57:12.861787 (+ 115us) write_op.cc:270] Start()
0112 08:57:12.861873 (+ 86us) write_op.cc:276] Timestamp: P: 1736672232861703 usec, L: 0
0112 08:57:12.861887 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 08:57:12.862387 (+ 500us) log.cc:844] Serialized 356 byte log entry
0112 08:57:12.945093 (+ 82706us) op_driver.cc:464] REPLICATION: finished
0112 08:57:12.969843 (+ 24750us) write_op.cc:301] APPLY: starting
0112 08:57:12.969911 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:12.970017 (+ 106us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:12.970237 (+ 220us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:12.970457 (+ 220us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:12.970663 (+ 206us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:12.970713 (+ 50us) write_op.cc:312] APPLY: finished
0112 08:57:12.970917 (+ 204us) log.cc:844] Serialized 23 byte log entry
0112 08:57:12.971089 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.971156 (+ 67us) write_op.cc:454] Released schema lock
0112 08:57:12.971189 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:12.971697 (+ 508us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.971725 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":47558,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":1143,"apply.run_wall_time_us":1143,"num_ops":1,"prepare.queue_time_us":55882,"prepare.run_cpu_time_us":2187,"prepare.run_wall_time_us":2188,"raft.queue_time_us":1301,"raft.run_cpu_time_us":1357,"raft.run_wall_time_us":1357,"replication_time_us":13638}],["op",{"apply.queue_time_us":24577,"apply.run_cpu_time_us":1434,"apply.run_wall_time_us":1759,"num_ops":1,"prepare.queue_time_us":34088,"prepare.run_cpu_time_us":1951,"prepare.run_wall_time_us":3036,"replication_time_us":83079}]]}
W20250112 08:57:13.199854 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 35f60d1f4bed4721b73deb9ba787417c because the tablet was not found. No further retry: Not found: Tablet not found: 35f60d1f4bed4721b73deb9ba787417c
I20250112 08:57:13.200995 22290 leader_election.cc:304] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.201668 23268 raft_consensus.cc:2798] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:13.201826 23312 ts_tablet_manager.cc:1918] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.202103 23268 raft_consensus.cc:695] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Becoming Leader. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Running, Role: LEADER
I20250112 08:57:13.202180 23312 log.cc:1198] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/5c3e34e1d70a4c63956a5be660e02b21
I20250112 08:57:13.203275 22357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e8f51e50994929b40a5e0679b07d8c" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:13.203802 22357 raft_consensus.cc:3054] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.206385 22327 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c6ea87589249739461e629eee5f40b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.206862 22336 tablet_service.cc:1514] Processing DeleteTablet for tablet 422e1484cd84479abba824cc8a6fae1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:07 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.207440 22325 tablet_service.cc:1514] Processing DeleteTablet for tablet a44cad3bd5a84e75a41733d8d2f345a8 with delete_type TABLET_DATA_DELETED (replaced by tablet e61e80e259db4645942141754c8491a3) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.206444 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1514074a19694aa0a42b968a29904c30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.208036 22327 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.208058 22335 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.208611 22323 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a22f87fbbf48e88dbd5af418ee94ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:01 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.209132 22324 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df6282224b849a488071323e2e69e1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:06 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.209719 22456 tablet_bootstrap.cc:492] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:13.210146 22456 ts_tablet_manager.cc:1397] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.004s
I20250112 08:57:13.210249 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e62435dce1441a3b2fee50518662d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:00 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.208225 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet 06717cb567a54ee6a268c3be5d8786ec with delete_type TABLET_DATA_DELETED (replaced by tablet db58dd0f600f403a91b54707703abd4b) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.202849 23268 consensus_queue.cc:237] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.214210 22357 raft_consensus.cc:2463] T 72e8f51e50994929b40a5e0679b07d8c P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:13.219815 22456 raft_consensus.cc:357] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.220517 22456 raft_consensus.cc:383] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.220759 22456 raft_consensus.cc:738] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.222329 22456 consensus_queue.cc:260] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
W20250112 08:57:13.223361 21714 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.194:55563 (request call id 48) took 1397 ms (client timeout 1000 ms). Trace:
I20250112 08:57:13.202828 23312 ts_tablet_manager.cc:1939] T 5c3e34e1d70a4c63956a5be660e02b21 P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.224704 22456 ts_tablet_manager.cc:1428] T 305aedf43e7241228ba88c90c329369d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20250112 08:57:13.226693 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 5c3e34e1d70a4c63956a5be660e02b21 (table default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]) successfully deleted
W20250112 08:57:13.182934 21713 rpcz_store.cc:269] 0112 08:57:11.679981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:12.123011 (+443030us) service_pool.cc:225] Handling call
0112 08:57:13.182539 (+1059528us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 08:57:12.826303 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:12.826848 (+ 545us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:12.826884 (+ 36us) write_op.cc:435] Acquired schema lock
0112 08:57:12.826909 (+ 25us) tablet.cc:659] Decoding operations
0112 08:57:12.848093 (+ 21184us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:12.848170 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:12.848197 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:12.848381 (+ 184us) tablet.cc:698] Row locks acquired
0112 08:57:12.848398 (+ 17us) write_op.cc:260] PREPARE: finished
0112 08:57:12.848531 (+ 133us) write_op.cc:270] Start()
0112 08:57:12.848623 (+ 92us) write_op.cc:276] Timestamp: P: 1736672232848443 usec, L: 0
0112 08:57:12.848641 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 08:57:12.849214 (+ 573us) log.cc:844] Serialized 533 byte log entry
0112 08:57:12.921304 (+ 72090us) op_driver.cc:464] REPLICATION: finished
0112 08:57:12.931314 (+ 10010us) write_op.cc:301] APPLY: starting
0112 08:57:12.931371 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:12.931460 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:12.931565 (+ 105us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:12.931749 (+ 184us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:12.931891 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:12.931973 (+ 82us) write_op.cc:312] APPLY: finished
0112 08:57:12.932199 (+ 226us) log.cc:844] Serialized 23 byte log entry
0112 08:57:12.932346 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.932410 (+ 64us) write_op.cc:454] Released schema lock
0112 08:57:12.932441 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:12.941590 (+ 9149us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:12.941615 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 08:57:12.960559 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 08:57:12.960975 (+ 416us) write_op.cc:432] Acquiring schema lock in shared mode
0112 08:57:12.961002 (+ 27us) write_op.cc:435] Acquired schema lock
0112 08:57:12.961019 (+ 17us) tablet.cc:659] Decoding operations
0112 08:57:12.961336 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0112 08:57:12.961372 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0112 08:57:12.961399 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0112 08:57:12.961552 (+ 153us) tablet.cc:698] Row locks acquired
0112 08:57:12.961573 (+ 21us) write_op.cc:260] PREPARE: finished
0112 08:57:12.961694 (+ 121us) write_op.cc:270] Start()
0112 08:57:12.961779 (+ 85us) write_op.cc:276] Timestamp: P: 1736672232961605 usec, L: 0
0112 08:57:12.961799 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 08:57:13.000138 (+ 38339us) log.cc:844] Serialized 356 byte log entry
0112 08:57:13.001930 (+ 1792us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf247526d1 00007fcf247c9198 00007fcf24c353b1 00007fcf24c34b88 00007fcf24c39821 00007fcf24c397d9 00007fcf24c39769 00007fcf24c39731 00007fcf24c3895d 000000000059ea04 000000000059e839
0112 08:57:13.010718 (+ 8788us) op_driver.cc:464] REPLICATION: finished
0112 08:57:13.011055 (+ 337us) write_op.cc:301] APPLY: starting
0112 08:57:13.011105 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 08:57:13.011194 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0112 08:57:13.011211 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 08:57:13.011424 (+ 213us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 08:57:13.011572 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 08:57:13.011613 (+ 41us) write_op.cc:312] APPLY: finished
0112 08:57:13.011802 (+ 189us) log.cc:844] Serialized 23 byte log entry
0112 08:57:13.011951 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:13.012011 (+ 60us) write_op.cc:454] Released schema lock
0112 08:57:13.012038 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0112 08:57:13.012258 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0112 08:57:13.012286 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":9839,"apply.run_cpu_time_us":1386,"apply.run_wall_time_us":10238,"num_ops":1,"prepare.queue_time_us":2399,"prepare.run_cpu_time_us":2591,"prepare.run_wall_time_us":34029,"raft.queue_time_us":7028,"raft.run_cpu_time_us":1457,"raft.run_wall_time_us":1457,"replication_time_us":72557,"thread_start_us":16574,"threads_started":2}],["op",{"apply.queue_time_us":160,"apply.run_cpu_time_us":1201,"apply.run_wall_time_us":1201,"num_ops":1,"prepare.queue_time_us":3607,"prepare.run_cpu_time_us":4379,"prepare.run_wall_time_us":41590,"raft.queue_time_us":1901,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":48822,"spinlock_wait_cycles":86781184}]]}
W20250112 08:57:13.230403 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet fc82026fcd1541418176a015ea3253cf because the tablet was not found. No further retry: Not found: Tablet not found: fc82026fcd1541418176a015ea3253cf
W20250112 08:57:13.231734 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet f46707de727a49108afb12ea2cf08aa0 because the tablet was not found. No further retry: Not found: Tablet not found: f46707de727a49108afb12ea2cf08aa0
I20250112 08:57:13.232579 23312 tablet_replica.cc:331] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.233429 23312 raft_consensus.cc:2238] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.233855 23312 raft_consensus.cc:2267] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.235590 23312 ts_tablet_manager.cc:1905] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.239629 22456 tablet_bootstrap.cc:492] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:13.243319 22461 tablet_bootstrap.cc:654] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:13.223683 21714 rpcz_store.cc:269] 0112 08:57:11.826176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:12.794982 (+968806us) service_pool.cc:225] Handling call
0112 08:57:13.223288 (+428306us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":34}
I20250112 08:57:13.261278 22456 tablet_bootstrap.cc:654] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:13.285209 21715 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.21.42.195:37593 (request call id 50) took 837 ms (client timeout 1000 ms). Trace:
W20250112 08:57:13.285547 21715 rpcz_store.cc:269] 0112 08:57:12.447347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:12.836149 (+388802us) service_pool.cc:225] Handling call
0112 08:57:12.852055 (+ 15906us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b54000316d4. stack: 00007fcf1d852853 00007fcf1d852750 00007fcf1d3c7584 00000000004fcd24 00000000004fccb9 00007fcf24752d50 00007fcf247d5f2a 00007fcf2647adec 00007fcf26572dee 00007fcf242d0fc4 00007fcf242d0f52 00007fcf242d0e81 00007fcf242d0dfc 00007fcf242d00b2 00007fcf2259adfc 00007fcf2259a236
0112 08:57:13.285151 (+433096us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41090,"reported_tablets":29,"spinlock_wait_cycles":3722496}
I20250112 08:57:13.287289 22461 tablet_bootstrap.cc:492] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.287751 22461 ts_tablet_manager.cc:1397] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.000s
I20250112 08:57:13.290087 22461 raft_consensus.cc:357] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.291131 22149 leader_election.cc:304] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.291803 23264 raft_consensus.cc:2798] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:13.292246 23264 raft_consensus.cc:695] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:13.292917 23264 consensus_queue.cc:237] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.293691 22461 raft_consensus.cc:383] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.293982 22461 raft_consensus.cc:738] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.297883 22456 tablet_bootstrap.cc:492] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:13.298300 22456 ts_tablet_manager.cc:1397] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.001s
I20250112 08:57:13.298038 22461 consensus_queue.cc:260] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.308821 23312 ts_tablet_manager.cc:1918] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.309230 23312 log.cc:1198] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/da16d10565a74243a9ab8c5264bf7b1a
I20250112 08:57:13.309880 23312 ts_tablet_manager.cc:1939] T da16d10565a74243a9ab8c5264bf7b1a P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.312433 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet da16d10565a74243a9ab8c5264bf7b1a (table default.Table_b4a3c3b699c348f98b156a1e95692b79 [id=f9fef379ee924c1c8fdc18e5b98fcb11]) successfully deleted
I20250112 08:57:13.312848 22332 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:57:13.313107 22290 leader_election.cc:343] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: fc82026fcd1541418176a015ea3253cf
I20250112 08:57:13.313448 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.314435 23188 raft_consensus.cc:491] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:13.314849 23188 raft_consensus.cc:513] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.316684 23188 leader_election.cc:290] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
I20250112 08:57:13.317104 23312 tablet_replica.cc:331] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.317905 23312 raft_consensus.cc:2238] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.319726 23312 raft_consensus.cc:2267] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:13.320590 21702 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d686b08555414954aba52ea077889660 consensus state has no leader
I20250112 08:57:13.321519 23312 ts_tablet_manager.cc:1905] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.331000 22461 ts_tablet_manager.cc:1428] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.043s user 0.004s sys 0.000s
I20250112 08:57:13.331816 22461 tablet_bootstrap.cc:492] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:13.339051 22290 leader_election.cc:343] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: da16d10565a74243a9ab8c5264bf7b1a
I20250112 08:57:13.345835 22461 tablet_bootstrap.cc:654] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.348529 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:13.349946 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 2 got 0
I20250112 08:57:13.350816 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:13.351320 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c7c48b95e2f24f8a8a2638d4df8ba3b8 (table default.Table_c7e1f3d4703d41ffb340c43933e0fd41 [id=d3dfc1c9cf0b4d52a2e152226ce5273e]). Expected version 1 got 0
I20250112 08:57:13.352527 22456 raft_consensus.cc:357] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.353252 22456 raft_consensus.cc:383] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.353524 22456 raft_consensus.cc:738] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.354082 22456 consensus_queue.cc:260] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } }
I20250112 08:57:13.363178 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet c228f04faa5c4f68a95adfb295235047 (table default.Table_b576158684104afd8673ccbe72b45c40 [id=929d59bdba3141f5947bc3af54185497]). Expected version 1 got 0
I20250112 08:57:13.363965 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:13.364872 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 57b5d9d5141941b69b33cc3f5d8b9ec0 (table default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2]). Expected version 2 got 0
I20250112 08:57:13.365891 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 340f93f630a34b3b9a9386999e2684e8 (table default.Table_1972c3e02af94e419ca99e14b22a365a [id=453899f3df2e41849bdc3cdd8c32b93b]). Expected version 1 got 0
I20250112 08:57:13.366354 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 74437c3dea6547f3b9545da3f8f971bb (table default.Table_d1591f699c154e6f9124d57a6779e410 [id=3d1f3e4fa4004b17b07b0978a27aff3f]). Expected version 1 got 0
I20250112 08:57:13.368225 22456 ts_tablet_manager.cc:1428] T 3b6bac902842453ba9d4ede5e8688f76 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20250112 08:57:13.368649 23312 ts_tablet_manager.cc:1918] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.369115 22456 tablet_bootstrap.cc:492] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:13.369199 23312 log.cc:1198] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/24e924f523524bd0a387597399e8470a
I20250112 08:57:13.370175 23312 ts_tablet_manager.cc:1939] T 24e924f523524bd0a387597399e8470a P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.359748 23103 ts_tablet_manager.cc:1918] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.370874 23103 log.cc:1198] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/0b38deeff1964b09938a12cfd4992ec5
I20250112 08:57:13.371507 23103 ts_tablet_manager.cc:1939] T 0b38deeff1964b09938a12cfd4992ec5 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:13.373816 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 24e924f523524bd0a387597399e8470a (table default.Table_0c1ffc62d2df4eeeba13b44b9a36a26d [id=bf87e713bdc94157a746f8e1d0aa877b]) successfully deleted
I20250112 08:57:13.379423 23264 raft_consensus.cc:491] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 08:57:13.380007 21701 catalog_manager.cc:4839] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): delete failed for tablet 6c34fc4b6aaf45fdae76cf153129f2da because the tablet was not found. No further retry: Not found: Tablet not found: 6c34fc4b6aaf45fdae76cf153129f2da
I20250112 08:57:13.379833 23264 raft_consensus.cc:513] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.380306 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 0b38deeff1964b09938a12cfd4992ec5 (table default.Table_97d2a91e67624174904c1f044679de1f [id=b49100d4fb5941a09fd68bba097bcbd7]) successfully deleted
I20250112 08:57:13.380940 23312 tablet_replica.cc:331] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.382202 23264 leader_election.cc:290] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:13.382284 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd228f7a432b436581368da7eab3ef16" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.382869 22355 raft_consensus.cc:2463] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 0.
I20250112 08:57:13.383708 22149 leader_election.cc:304] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.384334 23264 raft_consensus.cc:2798] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:13.384603 23103 tablet_replica.cc:331] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.384626 23264 raft_consensus.cc:491] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:13.384912 23264 raft_consensus.cc:3054] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.385380 23103 raft_consensus.cc:2238] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.385805 23103 raft_consensus.cc:2267] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.388033 23103 ts_tablet_manager.cc:1905] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.366999 21709 catalog_manager.cc:5526] T 98ddac23210248e3be1aba6591641b2d P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:13.391196 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.391358 23264 raft_consensus.cc:513] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.391083 23312 raft_consensus.cc:2238] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.392637 23312 raft_consensus.cc:2267] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.394439 23312 ts_tablet_manager.cc:1905] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.394913 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd228f7a432b436581368da7eab3ef16" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd"
I20250112 08:57:13.395248 22456 tablet_bootstrap.cc:654] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.395493 22355 raft_consensus.cc:3054] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:13.401736 22355 raft_consensus.cc:2463] T fd228f7a432b436581368da7eab3ef16 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53068c99960647728141706e9c50b870 in term 1.
I20250112 08:57:13.402909 22149 leader_election.cc:304] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53068c99960647728141706e9c50b870, 8c608785d6874abf9cc28db11a979ddd; no voters:
I20250112 08:57:13.403652 23335 raft_consensus.cc:2798] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:13.404098 23335 raft_consensus.cc:695] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [term 1 LEADER]: Becoming Leader. State: Replica: 53068c99960647728141706e9c50b870, State: Running, Role: LEADER
I20250112 08:57:13.404860 23335 consensus_queue.cc:237] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.418920 23268 raft_consensus.cc:491] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:13.421340 22456 tablet_bootstrap.cc:492] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:13.419739 23268 raft_consensus.cc:513] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.421732 22456 ts_tablet_manager.cc:1397] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.006s
I20250112 08:57:13.422961 23268 leader_election.cc:290] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76718d1596134c83900541c6d398d59e (127.21.42.193:37233), 53068c99960647728141706e9c50b870 (127.21.42.194:34135)
W20250112 08:57:13.425905 22290 leader_election.cc:343] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Not found: Tablet not found: 92a22f87fbbf48e88dbd5af418ee94ad
I20250112 08:57:13.427529 22461 tablet_bootstrap.cc:492] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.427942 22461 ts_tablet_manager.cc:1397] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.000s
I20250112 08:57:13.432328 22461 raft_consensus.cc:357] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.432972 22461 raft_consensus.cc:383] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.433236 22461 raft_consensus.cc:738] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.434809 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.435447 21712 catalog_manager.cc:5526] T d686b08555414954aba52ea077889660 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:13.436645 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet d686b08555414954aba52ea077889660 (table default.Table_f6b0db0e8c6a4dea88649d3dc4286a81 [id=efadbdd973ee4d919b3018f578ff7dbd]). Expected version 2 got 0
I20250112 08:57:13.437647 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:13.442704 22461 consensus_queue.cc:260] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.438266 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:13.449178 21712 catalog_manager.cc:5526] T 05401e13af414ce9bf8fdb6fce76e254 P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:13.450642 21712 catalog_manager.cc:5526] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c608785d6874abf9cc28db11a979ddd (127.21.42.195). New cstate: current_term: 1 leader_uuid: "8c608785d6874abf9cc28db11a979ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } }
I20250112 08:57:13.428328 23264 leader_election.cc:290] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 election: Requested vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:13.423885 22456 raft_consensus.cc:357] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.464331 22456 raft_consensus.cc:383] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.464618 22456 raft_consensus.cc:738] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.465304 22456 consensus_queue.cc:260] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.467033 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet fc82026fcd1541418176a015ea3253cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.468032 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3e34e1d70a4c63956a5be660e02b21 with delete_type TABLET_DATA_DELETED (replaced by tablet 9d669b1a721a41f9b9902c57ee8c33aa) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.468851 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.469743 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet f46707de727a49108afb12ea2cf08aa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:04 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.473079 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet c49ec037c22a42108ecb48bc8270191f with delete_type TABLET_DATA_DELETED (replaced by tablet b7c3c7162ff144a79715944b7c9f0371) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.473999 22187 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.476425 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e924f523524bd0a387597399e8470a with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:37418
I20250112 08:57:13.479578 22461 ts_tablet_manager.cc:1428] T 6368a80a9fe9457a803e68a2392b638d P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.051s user 0.002s sys 0.003s
I20250112 08:57:13.483893 22456 ts_tablet_manager.cc:1428] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.062s user 0.006s sys 0.001s
I20250112 08:57:13.484802 22456 tablet_bootstrap.cc:492] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:13.490468 23312 ts_tablet_manager.cc:1918] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.492564 23312 log.cc:1198] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/54b34b09f56d4417a3f321599170e5fd
I20250112 08:57:13.493759 23312 ts_tablet_manager.cc:1939] T 54b34b09f56d4417a3f321599170e5fd P 53068c99960647728141706e9c50b870: Deleting consensus metadata
I20250112 08:57:13.497593 23103 ts_tablet_manager.cc:1918] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.497952 23103 log.cc:1198] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/ea15d9405b9944db8e00121965b9ffde
I20250112 08:57:13.498572 23103 ts_tablet_manager.cc:1939] T ea15d9405b9944db8e00121965b9ffde P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:13.500785 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet ea15d9405b9944db8e00121965b9ffde (table default.Table_250ba1babd9c419e8933849397cdff93 [id=a42ff5d5b1f949ef97f131233f826798]) successfully deleted
I20250112 08:57:13.501163 23103 tablet_replica.cc:331] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
W20250112 08:57:13.501482 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 5cd402674080464cb3bde3697ac23180 because the tablet was not found. No further retry: Not found: Tablet not found: 5cd402674080464cb3bde3697ac23180
I20250112 08:57:13.501868 23103 raft_consensus.cc:2238] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.502259 23103 raft_consensus.cc:2267] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.508031 23335 raft_consensus.cc:491] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:13.510478 23335 raft_consensus.cc:513] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.512346 23335 leader_election.cc:290] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805), 76718d1596134c83900541c6d398d59e (127.21.42.193:37233)
I20250112 08:57:13.510092 22456 tablet_bootstrap.cc:654] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.510933 21712 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 15d38686b2c64b20ba87275e3ba176ab: default.Table_5bd9277e65f04760b8979a42ceef3074 [id=a2eefae2a7d9415ba1ca7fb01af9ebe2] alter complete (version 2)
I20250112 08:57:13.516628 21701 catalog_manager.cc:4872] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135): tablet 54b34b09f56d4417a3f321599170e5fd (table default.Table_794e751834834ecebf5515268c2a7eb1 [id=7838ea67766144e094e6392661d918bc]) successfully deleted
I20250112 08:57:13.517422 22355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6241252631a4646812ed6ecfbf7ce6d" candidate_uuid: "53068c99960647728141706e9c50b870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c608785d6874abf9cc28db11a979ddd" is_pre_election: true
I20250112 08:57:13.518921 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet f9835288a7974e85821b095c03e91886 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.519608 23312 tablet_replica.cc:331] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: stopping tablet replica
I20250112 08:57:13.519959 23103 ts_tablet_manager.cc:1905] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.522184 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c34fc4b6aaf45fdae76cf153129f2da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:02 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.523666 22461 tablet_bootstrap.cc:492] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
W20250112 08:57:13.519629 22149 leader_election.cc:343] T f6241252631a4646812ed6ecfbf7ce6d P 53068c99960647728141706e9c50b870 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:13.527980 23312 raft_consensus.cc:2238] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.528488 23312 raft_consensus.cc:2267] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.541392 23312 ts_tablet_manager.cc:1905] T f9835288a7974e85821b095c03e91886 P 53068c99960647728141706e9c50b870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.545249 22461 tablet_bootstrap.cc:654] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:13.549408 22290 consensus_peers.cc:487] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd -> Peer 53068c99960647728141706e9c50b870 (127.21.42.194:34135): Couldn't send request to peer 53068c99960647728141706e9c50b870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c3e34e1d70a4c63956a5be660e02b21. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:13.554774 21709 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.558022 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet da16d10565a74243a9ab8c5264bf7b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:05 UTC) from {username='slave'} at 127.0.0.1:44720
I20250112 08:57:13.560823 22456 tablet_bootstrap.cc:492] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870: No bootstrap required, opened a new log
I20250112 08:57:13.561383 22456 ts_tablet_manager.cc:1397] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.005s
I20250112 08:57:13.564038 22456 raft_consensus.cc:357] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.565043 22456 raft_consensus.cc:383] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.565352 22456 raft_consensus.cc:738] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53068c99960647728141706e9c50b870, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.566105 22456 consensus_queue.cc:260] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.568564 22456 ts_tablet_manager.cc:1428] T 591040f7b9f9451a9979f51b2721b7a4 P 53068c99960647728141706e9c50b870: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250112 08:57:13.585973 21709 catalog_manager.cc:5526] T 72e8f51e50994929b40a5e0679b07d8c P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:13.587491 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 53167910ab0a4accb1d1615d537ba7ae (table default.Table_66ec2be61ba0470195e2c535112eb097 [id=1b9b675047ab4ad3a5b0d3028a578303]). Expected version 2 got 0
I20250112 08:57:13.588027 22456 tablet_bootstrap.cc:492] T bbdd48de9ed74a788694e9b9ba6e3fff P 53068c99960647728141706e9c50b870: Bootstrap starting.
I20250112 08:57:13.588145 21709 catalog_manager.cc:5526] T fd228f7a432b436581368da7eab3ef16 P 53068c99960647728141706e9c50b870 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53068c99960647728141706e9c50b870 (127.21.42.194). New cstate: current_term: 1 leader_uuid: "53068c99960647728141706e9c50b870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:13.589264 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet fd228f7a432b436581368da7eab3ef16 (table default.Table_e61de8eab47c47008e26794edc339a29 [id=960814982a0b4f6bbb3d2032137bdd1f]). Expected version 1 got 0
I20250112 08:57:13.590142 21709 catalog_manager.cc:5618] TS 53068c99960647728141706e9c50b870 (127.21.42.194:34135) does not have the latest schema for tablet 98ddac23210248e3be1aba6591641b2d (table default.Table_60490ab3bd724ea8bb2f2f573b901938 [id=dc7a2e8940134ac184a6434f029cd95d]). Expected version 1 got 0
I20250112 08:57:13.559692 23103 ts_tablet_manager.cc:1918] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.592417 23103 log.cc:1198] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/80c6ea87589249739461e629eee5f40b
I20250112 08:57:13.593065 23103 ts_tablet_manager.cc:1939] T 80c6ea87589249739461e629eee5f40b P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:13.618001 22461 tablet_bootstrap.cc:492] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.618445 22461 ts_tablet_manager.cc:1397] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.116s user 0.016s sys 0.003s
I20250112 08:57:13.621603 22461 raft_consensus.cc:357] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:13.622282 22461 raft_consensus.cc:383] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.622506 22461 raft_consensus.cc:738] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.623080 22461 consensus_queue.cc:260] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } } peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } }
I20250112 08:57:13.625065 22461 ts_tablet_manager.cc:1428] T e61e80e259db4645942141754c8491a3 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250112 08:57:13.626346 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 80c6ea87589249739461e629eee5f40b (table default.Table_bda308e111924297bc1d064e44647a8c [id=78bc9e684bde43269ebdb804dd657e65]) successfully deleted
I20250112 08:57:13.629045 22456 tablet_bootstrap.cc:654] T bbdd48de9ed74a788694e9b9ba6e3fff P 53068c99960647728141706e9c50b870: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.632625 22461 tablet_bootstrap.cc:492] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:13.633934 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fc4d2d0eaa47f6a02f518828253057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:08 UTC) from {username='slave'} at 127.0.0.1:37418
W20250112 08:57:13.642427 21701 connection.cc:537] client connection to 127.21.42.193:37233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:13.642791 21701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:13.642964 22146 connection.cc:537] client connection to 127.21.42.193:37233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:13.643404 22146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 08:57:13.648484 22337 tablet_service.cc:1514] Processing DeleteTablet for tablet e61e80e259db4645942141754c8491a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 08:57:09 UTC) from {username='slave'} at 127.0.0.1:44720
W20250112 08:57:13.649046 22290 connection.cc:537] client connection to 127.21.42.193:37233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:13.649471 22290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 08:57:13.662163 23103 tablet_replica.cc:331] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.662925 23103 raft_consensus.cc:2238] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: Raft consensus shutting down.
I20250112 08:57:13.663594 23103 pending_rounds.cc:62] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Trying to abort 1 pending ops.
I20250112 08:57:13.672690 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 22120
W20250112 08:57:13.676000 21701 connection.cc:537] server connection from 127.21.42.193:36265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 08:57:13.663823 23103 pending_rounds.cc:69] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409463514050560 op_type: NO_OP noop_request { }
I20250112 08:57:13.681097 23103 raft_consensus.cc:2883] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:13.681478 23103 raft_consensus.cc:2267] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.683336 23103 ts_tablet_manager.cc:1905] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.686877 22461 tablet_bootstrap.cc:654] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:13.693269 21701 catalog_manager.cc:4615] TS 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e61e80e259db4645942141754c8491a3: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
W20250112 08:57:13.697057 22290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.42.193:37233, user_credentials={real_user=slave}} blocked reactor thread for 47639.2us
W20250112 08:57:13.704375 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet 422e1484cd84479abba824cc8a6fae1f because the tablet was not found. No further retry: Not found: Tablet not found: 422e1484cd84479abba824cc8a6fae1f
W20250112 08:57:13.705432 21704 catalog_manager.cc:4839] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): delete failed for tablet a44cad3bd5a84e75a41733d8d2f345a8 because the tablet was not found. No further retry: Not found: Tablet not found: a44cad3bd5a84e75a41733d8d2f345a8
I20250112 08:57:13.718469 22461 tablet_bootstrap.cc:492] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.718888 22461 ts_tablet_manager.cc:1397] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.005s
I20250112 08:57:13.721122 22461 raft_consensus.cc:357] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.721802 22461 raft_consensus.cc:383] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.722038 22461 raft_consensus.cc:738] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.722602 22461 consensus_queue.cc:260] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.729769 23103 ts_tablet_manager.cc:1918] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:13.730188 23103 log.cc:1198] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/5c3e34e1d70a4c63956a5be660e02b21
I20250112 08:57:13.730891 23103 ts_tablet_manager.cc:1939] T 5c3e34e1d70a4c63956a5be660e02b21 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
W20250112 08:57:13.736088 21748 debug-util.cc:398] Leaking SignalData structure 0x7b08000c18e0 after lost signal to thread 21685
W20250112 08:57:13.736860 21748 debug-util.cc:398] Leaking SignalData structure 0x7b0800054b20 after lost signal to thread 21751
W20250112 08:57:13.745174 22290 leader_election.cc:336] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
W20250112 08:57:13.745887 22290 consensus_peers.cc:487] T 815d246b7d414aadbb69ac692a5d40c7 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:13.746290 22290 consensus_peers.cc:487] T 003f1f1414584353a86214d27401291e P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:13.746724 22290 leader_election.cc:336] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
W20250112 08:57:13.747314 22290 consensus_peers.cc:487] T 57b5d9d5141941b69b33cc3f5d8b9ec0 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:13.747659 22290 consensus_peers.cc:487] T 02538a000a0f4fa5a40f1ab5cd862fe1 P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:13.756122 22290 consensus_peers.cc:487] T f61aedea18d944179e788c08488c0a5e P 8c608785d6874abf9cc28db11a979ddd -> Peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Couldn't send request to peer 76718d1596134c83900541c6d398d59e. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 08:57:13.756562 22290 leader_election.cc:336] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
I20250112 08:57:13.756929 22290 leader_election.cc:304] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:13.757642 23268 raft_consensus.cc:2743] T f46707de727a49108afb12ea2cf08aa0 P 8c608785d6874abf9cc28db11a979ddd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250112 08:57:13.757768 22290 leader_election.cc:336] T ef9c33e005b645e29464178c92a2466a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
W20250112 08:57:13.758411 22290 leader_election.cc:336] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
I20250112 08:57:13.758764 22290 leader_election.cc:304] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:13.759370 23268 raft_consensus.cc:2743] T fc82026fcd1541418176a015ea3253cf P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:13.759516 22290 leader_election.cc:336] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
I20250112 08:57:13.759923 22290 leader_election.cc:304] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:13.760519 23268 raft_consensus.cc:2743] T da16d10565a74243a9ab8c5264bf7b1a P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:13.760768 22290 leader_election.cc:336] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 76718d1596134c83900541c6d398d59e (127.21.42.193:37233): Network error: Client connection negotiation failed: client connection to 127.21.42.193:37233: connect: Connection refused (error 111)
I20250112 08:57:13.761137 22290 leader_election.cc:304] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c608785d6874abf9cc28db11a979ddd; no voters: 53068c99960647728141706e9c50b870, 76718d1596134c83900541c6d398d59e
I20250112 08:57:13.761704 23268 raft_consensus.cc:2743] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:13.762310 22461 ts_tablet_manager.cc:1428] T b62bae16ab43414c8d8be2766802d4c7 P 8c608785d6874abf9cc28db11a979ddd: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20250112 08:57:13.762612 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 5c3e34e1d70a4c63956a5be660e02b21 (table default.Table_1494ad860b76492cbcbb83c5341729dc [id=934be1009a23491798fde07ccd2daa4c]) successfully deleted
I20250112 08:57:13.749421 23103 tablet_replica.cc:331] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.763691 23103 raft_consensus.cc:2238] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.764693 23103 raft_consensus.cc:2267] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.766335 23103 ts_tablet_manager.cc:1905] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.767717 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:13.790434 22461 tablet_bootstrap.cc:492] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd: Bootstrap starting.
I20250112 08:57:13.802692 21712 catalog_manager.cc:5618] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805) does not have the latest schema for tablet b62bae16ab43414c8d8be2766802d4c7 (table default.Table_72e464e993c04f199942af992f07f98d [id=e35c684b420f477e8e8fb3b9f8f058f5]). Expected version 2 got 0
I20250112 08:57:13.807773 23103 ts_tablet_manager.cc:1918] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.808203 23103 log.cc:1198] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/ac1889a1ab4f4d8aa4f7915d91f2cf27
I20250112 08:57:13.808825 23103 ts_tablet_manager.cc:1939] T ac1889a1ab4f4d8aa4f7915d91f2cf27 P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:13.822278 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet ac1889a1ab4f4d8aa4f7915d91f2cf27 (table default.Table_da91c42fe80845d5b255227302fa7244 [id=5a31974b079c4baca492c04d4875dcaf]) successfully deleted
I20250112 08:57:13.823246 23103 tablet_replica.cc:331] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.828751 23103 raft_consensus.cc:2238] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.829274 23103 raft_consensus.cc:2267] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.835736 22461 tablet_bootstrap.cc:654] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:13.840337 23103 ts_tablet_manager.cc:1905] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:13.888451 23103 ts_tablet_manager.cc:1918] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:13.888826 23103 log.cc:1198] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/92a22f87fbbf48e88dbd5af418ee94ad
I20250112 08:57:13.889459 23103 ts_tablet_manager.cc:1939] T 92a22f87fbbf48e88dbd5af418ee94ad P 8c608785d6874abf9cc28db11a979ddd: Deleting consensus metadata
I20250112 08:57:13.889640 22461 tablet_bootstrap.cc:492] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd: No bootstrap required, opened a new log
I20250112 08:57:13.890053 22461 ts_tablet_manager.cc:1397] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd: Time spent bootstrapping tablet: real 0.127s user 0.008s sys 0.001s
I20250112 08:57:13.898594 21704 catalog_manager.cc:4872] TS 8c608785d6874abf9cc28db11a979ddd (127.21.42.195:33805): tablet 92a22f87fbbf48e88dbd5af418ee94ad (table default.Table_ec36ed57a80f4350bda6894141741ba3 [id=6cb07113f32f4f039b2f200179a14796]) successfully deleted
W20250112 08:57:13.899699 21701 connection.cc:537] server connection from 127.21.42.194:55563 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 08:57:13.900094 22461 raft_consensus.cc:357] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
W20250112 08:57:13.900588 22290 connection.cc:537] client connection to 127.21.42.194:34135 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 08:57:13.900777 22461 raft_consensus.cc:383] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:13.901052 22461 raft_consensus.cc:738] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c608785d6874abf9cc28db11a979ddd, State: Initialized, Role: FOLLOWER
I20250112 08:57:13.901600 22461 consensus_queue.cc:260] T 54b34b09f56d4417a3f321599170e5fd P 8c608785d6874abf9cc28db11a979ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c608785d6874abf9cc28db11a979ddd" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 33805 } } peers { permanent_uuid: "76718d1596134c83900541c6d398d59e" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 37233 } } peers { permanent_uuid: "53068c99960647728141706e9c50b870" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 34135 } }
I20250112 08:57:13.914697 23103 tablet_replica.cc:331] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: stopping tablet replica
I20250112 08:57:13.915483 23103 raft_consensus.cc:2238] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:13.915973 23103 raft_consensus.cc:2267] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:13.917583 23103 ts_tablet_manager.cc:1905] T 1df6282224b849a488071323e2e69e1c P 8c608785d6874abf9cc28db11a979ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 08:57:13.921777 23355 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 08:57:13.904246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.42.194:34135 (local address 127.21.42.195:37493)
0112 08:57:13.905407 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0112 08:57:13.905476 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0112 08:57:13.905899 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 08:57:13.920964 (+ 15065us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.42.194:34135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":578}
I20250112 08:57:13.922519 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 22271
W20250112 08:57:13.927477 23353 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 08:57:13.909741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.42.194:34135 (local address 127.0.0.1:57688)
0112 08:57:13.910361 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0112 08:57:13.910402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0112 08:57:13.910861 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 08:57:13.926425 (+ 15564us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.42.194:34135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":64}
W20250112 08:57:14.024683 21701 connection.cc:537] server connection from 127.21.42.195:37593 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:14.026356 23358 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 08:57:14.022276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.42.195:33805 (local address 127.0.0.1:32788)
0112 08:57:14.023083 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0112 08:57:14.023124 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0112 08:57:14.023381 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 08:57:14.025470 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.42.195:33805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":263}
I20250112 08:57:14.029517 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 21684
I20250112 08:57:14.103955 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 21754
I20250112 08:57:14.113636 23359 raft_consensus.cc:491] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15d38686b2c64b20ba87275e3ba176ab)
I20250112 08:57:14.114077 23359 raft_consensus.cc:513] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d38686b2c64b20ba87275e3ba176ab" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 45457 } } peers { permanent_uuid: "5fc0c669abed404c80d7c4cf4769f55a" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 41411 } } peers { permanent_uuid: "caa33dde8b5248e7ab4d6087d8e9d801" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 43161 } }
I20250112 08:57:14.115659 23359 leader_election.cc:290] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457), 5fc0c669abed404c80d7c4cf4769f55a (127.21.42.253:41411)
W20250112 08:57:14.118355 22818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.254:45457: connect: Connection refused (error 111)
W20250112 08:57:14.121503 22818 leader_election.cc:336] T 00000000000000000000000000000000 P caa33dde8b5248e7ab4d6087d8e9d801 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 15d38686b2c64b20ba87275e3ba176ab (127.21.42.254:45457): Network error: Client connection negotiation failed: client connection to 127.21.42.254:45457: connect: Connection refused (error 111)
I20250112 08:57:14.157217 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 22572
2025-01-12T08:57:14Z chronyd exiting
I20250112 08:57:14.209731 21675 master-stress-test.cc:585] Tables created: 140
I20250112 08:57:14.209924 21675 master-stress-test.cc:586] Tables altered: 70
I20250112 08:57:14.210106 21675 master-stress-test.cc:587] Tables deleted: 59
I20250112 08:57:14.210271 21675 master-stress-test.cc:588] Tablets replaced: 15
I20250112 08:57:14.210420 21675 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (36518 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250112 08:57:14.280817 21675 test_util.cc:274] Using random seed: -2086824102
2025-01-12T08:57:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T08:57:14Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskTU_TaI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskTU_TaI/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/security.properties
2025-01-12 08:57:20: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskTU_TaI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskTU_TaI/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250112 08:57:29.037175 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:41739
--webserver_interface=127.21.42.254
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--hive_metastore_uris=thrift://127.0.0.1:39997
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 08:57:29.343513 23600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:29.344172 23600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:29.344610 23600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:29.376055 23600 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 08:57:29.376325 23600 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 08:57:29.376550 23600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:57:29.376775 23600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:29.377039 23600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:57:29.377262 23600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:57:29.412712 23600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39997
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.42.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:29.414093 23600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:29.415628 23600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:29.429873 23606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:29.430208 23607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:29.431134 23609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:29.432400 23600 server_base.cc:1034] running on GCE node
I20250112 08:57:30.606969 23600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:30.610114 23600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:30.611546 23600 hybrid_clock.cc:648] HybridClock initialized: now 1736672250611505 us; error 59 us; skew 500 ppm
I20250112 08:57:30.612409 23600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:30.619127 23600 webserver.cc:458] Webserver started at http://127.21.42.254:42815/ using document root <none> and password file <none>
I20250112 08:57:30.620141 23600 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:30.620348 23600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:30.620797 23600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:30.625824 23600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/instance:
uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
format_stamp: "Formatted at 2025-01-12 08:57:30 on dist-test-slave-ghv6"
I20250112 08:57:30.626952 23600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal/instance:
uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
format_stamp: "Formatted at 2025-01-12 08:57:30 on dist-test-slave-ghv6"
I20250112 08:57:30.634317 23600 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250112 08:57:30.639808 23616 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:30.640919 23600 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250112 08:57:30.641263 23600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
format_stamp: "Formatted at 2025-01-12 08:57:30 on dist-test-slave-ghv6"
I20250112 08:57:30.641593 23600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:30.697571 23600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:30.699025 23600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:30.699486 23600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:30.769379 23600 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.254:41739
I20250112 08:57:30.769484 23667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.254:41739 every 8 connection(s)
I20250112 08:57:30.772150 23600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
I20250112 08:57:30.774116 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 23600
I20250112 08:57:30.774488 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal/instance
I20250112 08:57:30.777027 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.253:36201
--webserver_interface=127.21.42.253
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--hive_metastore_uris=thrift://127.0.0.1:39997
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:57:30.777312 23668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:30.788159 23668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:30.815062 23668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:57:30.817579 23620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:30.821664 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:30.871230 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 1
W20250112 08:57:30.875082 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:30.943459 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 2
W20250112 08:57:30.947458 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:31.038864 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 3
W20250112 08:57:31.042827 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:31.088320 23669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:31.088912 23669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:31.089382 23669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:31.125201 23669 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 08:57:31.125454 23669 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 08:57:31.125705 23669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:57:31.125948 23669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:31.126174 23669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:57:31.126399 23669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:57:31.162379 23669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39997
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.253:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.42.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:31.163776 23669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:31.165377 23669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:31.181223 23678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:31.186177 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 4
W20250112 08:57:31.190237 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:31.489660 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 5
W20250112 08:57:31.494278 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:32.041656 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 6
W20250112 08:57:32.048475 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:31.182026 23679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:31.182535 23669 server_base.cc:1034] running on GCE node
W20250112 08:57:31.182904 23681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:32.366909 23669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:32.370047 23669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:32.371469 23669 hybrid_clock.cc:648] HybridClock initialized: now 1736672252371429 us; error 58 us; skew 500 ppm
I20250112 08:57:32.372500 23669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:32.379336 23669 webserver.cc:458] Webserver started at http://127.21.42.253:43113/ using document root <none> and password file <none>
I20250112 08:57:32.380352 23669 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:32.380581 23669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:32.381039 23669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:32.386094 23669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data/instance:
uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
format_stamp: "Formatted at 2025-01-12 08:57:32 on dist-test-slave-ghv6"
I20250112 08:57:32.387238 23669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal/instance:
uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
format_stamp: "Formatted at 2025-01-12 08:57:32 on dist-test-slave-ghv6"
I20250112 08:57:32.394537 23669 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250112 08:57:32.400182 23690 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:32.401310 23669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250112 08:57:32.401640 23669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal
uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
format_stamp: "Formatted at 2025-01-12 08:57:32 on dist-test-slave-ghv6"
I20250112 08:57:32.402000 23669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:32.459419 23669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:32.460959 23669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:32.461488 23669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:32.530251 23669 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.253:36201
I20250112 08:57:32.530331 23741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.253:36201 every 8 connection(s)
I20250112 08:57:32.533042 23669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/data/info.pb
I20250112 08:57:32.538203 23742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:32.538420 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 23669
I20250112 08:57:32.538901 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-1/wal/instance
I20250112 08:57:32.542485 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:36643
--webserver_interface=127.21.42.252
--webserver_port=0
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--hive_metastore_uris=thrift://127.0.0.1:39997
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:57:32.554046 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:32.575554 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:32.590040 23742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:57:32.592131 23694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:32.596305 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:32.645903 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 1
W20250112 08:57:32.649896 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:32.718410 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 2
W20250112 08:57:32.722334 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:32.813798 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 3
W20250112 08:57:32.817505 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:32.872040 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:32.872574 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:32.872978 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:32.904088 23743 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 08:57:32.904330 23743 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 08:57:32.904522 23743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:57:32.904712 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:32.904994 23743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:57:32.905217 23743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:57:32.940589 23743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39997
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.42.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:32.941887 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:32.943410 23743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:32.957464 23754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:32.960961 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 4
W20250112 08:57:32.965121 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.109138 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } attempt: 7
I20250112 08:57:33.127153 23668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } has no permanent_uuid. Determining permanent_uuid...
W20250112 08:57:33.133481 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.191876 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 1
W20250112 08:57:33.195595 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.264513 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 5
W20250112 08:57:33.270449 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.277261 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 2
W20250112 08:57:33.281303 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.366879 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 3
W20250112 08:57:33.372071 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.512504 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 4
W20250112 08:57:33.517675 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.801079 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 5
W20250112 08:57:33.806175 23668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:33.817906 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 6
W20250112 08:57:33.825839 23742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.42.252:36643: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:32.957737 23753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:34.144019 23755 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250112 08:57:34.145735 23756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:34.145794 23743 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 08:57:34.148850 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:34.151469 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:34.152818 23743 hybrid_clock.cc:648] HybridClock initialized: now 1736672254152751 us; error 71 us; skew 500 ppm
I20250112 08:57:34.153568 23743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:34.159498 23743 webserver.cc:458] Webserver started at http://127.21.42.252:38649/ using document root <none> and password file <none>
I20250112 08:57:34.160511 23743 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:34.160708 23743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:34.161150 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:34.166343 23743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/instance:
uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
format_stamp: "Formatted at 2025-01-12 08:57:34 on dist-test-slave-ghv6"
I20250112 08:57:34.167460 23743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal/instance:
uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
format_stamp: "Formatted at 2025-01-12 08:57:34 on dist-test-slave-ghv6"
I20250112 08:57:34.174711 23743 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250112 08:57:34.180090 23768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:34.181104 23743 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250112 08:57:34.181420 23743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
format_stamp: "Formatted at 2025-01-12 08:57:34 on dist-test-slave-ghv6"
I20250112 08:57:34.181741 23743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:34.231014 23743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:34.232491 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:34.232924 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:34.300923 23743 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.252:36643
I20250112 08:57:34.300982 23819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.252:36643 every 8 connection(s)
I20250112 08:57:34.303689 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
I20250112 08:57:34.305665 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 23743
I20250112 08:57:34.306062 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal/instance
I20250112 08:57:34.309247 23820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:34.329967 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:34.352047 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:34.358561 23668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 6
I20250112 08:57:34.363070 23820 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } has no permanent_uuid. Determining permanent_uuid...
I20250112 08:57:34.379244 23668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Bootstrap starting.
I20250112 08:57:34.381425 23820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Bootstrap starting.
I20250112 08:57:34.385524 23668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:34.388042 23668 log.cc:826] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:34.388669 23820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:34.390398 23820 log.cc:826] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:34.392505 23668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: No bootstrap required, opened a new log
I20250112 08:57:34.394850 23820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: No bootstrap required, opened a new log
I20250112 08:57:34.415493 23668 raft_consensus.cc:357] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.416385 23668 raft_consensus.cc:383] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:34.416702 23668 raft_consensus.cc:738] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bb7660e3b04a9ca5fb697b25b17bbc, State: Initialized, Role: FOLLOWER
I20250112 08:57:34.416514 23820 raft_consensus.cc:357] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.417176 23820 raft_consensus.cc:383] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:34.417407 23820 raft_consensus.cc:738] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Initialized, Role: FOLLOWER
I20250112 08:57:34.417532 23668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.418147 23820 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.420207 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.421178 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:34.421092 23831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.421962 23831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:34.422173 23668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:57:34.422866 23820 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:57:34.449512 23831 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:34.450218 23831 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.454697 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:34.455027 23831 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
I20250112 08:57:34.455708 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 0.
I20250112 08:57:34.457943 23769 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters:
W20250112 08:57:34.459210 23622 tablet.cc:2367] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:57:34.460114 23831 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:34.460539 23831 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:34.460942 23831 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:34.478152 23831 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:34.483942 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
I20250112 08:57:34.484747 23643 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:34.488317 23774 tablet.cc:2367] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:57:34.500434 23831 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 1 election: Requested vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
I20250112 08:57:34.512523 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 1.
I20250112 08:57:34.516327 23769 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters:
I20250112 08:57:34.517252 23831 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:34.535039 23831 raft_consensus.cc:695] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 LEADER]: Becoming Leader. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Running, Role: LEADER
I20250112 08:57:34.536121 23831 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
W20250112 08:57:34.579149 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.580864 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.610558 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.610925 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.611781 23859 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.613013 23859 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
W20250112 08:57:34.625614 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.625970 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.630632 23864 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.630982 23864 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.635782 23865 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250112 08:57:34.640518 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.640861 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.655295 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.655623 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.528224 23600 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: OK, instance_id { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" instance_seqno: 1736672252511433 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 08:57:34.660172 23600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:57:34.565452 23743 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: OK, instance_id { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" instance_seqno: 1736672252511433 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 08:57:34.663821 23743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:57:34.669965 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.670297 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.685204 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.685539 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.689800 23865 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250112 08:57:34.701021 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.701354 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.715926 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.716275 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.730744 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.731092 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.743732 23643 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Refusing update from remote peer e7c2dbf455f04f4fbcc282c86eae73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:57:34.745949 23859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:34.746538 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.746869 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.778716 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.779502 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
W20250112 08:57:34.784430 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 08:57:34.784783 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.791914 23867 mvcc.cc:204] Tried to move back new op lower bound from 7113409555406811136 to 7113409554591244288. Current Snapshot: MvccSnapshot[applied={T|T < 7113409555406811136}]
W20250112 08:57:34.807044 23857 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250112 08:57:34.808025 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.809351 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
W20250112 08:57:34.809911 23857 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.817677 23831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:34.818516 23831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:34.826063 23857 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:34.826412 23857 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: success
I20250112 08:57:34.828589 23865 catalog_manager.cc:1348] Generated new cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:34.828892 23865 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250112 08:57:34.831142 23857 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250112 08:57:34.833722 23857 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.849229 23857 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250112 08:57:34.850703 23857 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.866072 23857 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250112 08:57:34.866495 23857 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250112 08:57:34.881742 23857 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250112 08:57:34.882179 23857 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.886188 23742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } attempt: 7
W20250112 08:57:34.897857 23857 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250112 08:57:34.898453 23857 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.913292 23865 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 08:57:34.918010 23865 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:34.930672 23857 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: success
W20250112 08:57:34.937242 23857 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.963954 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Bootstrap starting.
W20250112 08:57:34.965482 23857 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250112 08:57:34.973410 23742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:34.975518 23865 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Generated new TSK 0
I20250112 08:57:34.978858 23865 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:34.987346 23865 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:34.990235 23742 log.cc:826] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:34.992825 23857 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:34.994786 23865 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 08:57:35.004050 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: No bootstrap required, opened a new log
I20250112 08:57:35.007931 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.193:0
--local_ip_for_outbound_sockets=127.21.42.193
--webserver_interface=127.21.42.193
--webserver_port=0
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:57:35.049697 23742 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:35.050714 23742 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:35.051093 23742 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6dc008c89249efbae18aa4e34b9ea8, State: Initialized, Role: FOLLOWER
I20250112 08:57:35.052151 23742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
W20250112 08:57:35.087757 23772 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.42.253:36201 timed out after 0.587s (SENT)
I20250112 08:57:35.099741 23876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:35.100628 23876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:35.101261 23742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:57:35.101415 23714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
I20250112 08:57:35.102279 23714 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:35.104876 23717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
W20250112 08:57:35.106940 23717 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39392 (request call id 1) took 652 ms (client timeout 651 ms). Trace:
I20250112 08:57:35.105854 23716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
I20250112 08:57:35.107728 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
W20250112 08:57:35.108343 23716 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39392 (request call id 2) took 608 ms (client timeout 587 ms). Trace:
W20250112 08:57:35.108659 23716 rpcz_store.cc:269] 0112 08:57:34.499775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:34.504002 (+ 4227us) service_pool.cc:225] Handling call
0112 08:57:35.108280 (+604278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 08:57:35.109054 23714 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 1.
W20250112 08:57:35.107329 23717 rpcz_store.cc:269] 0112 08:57:34.454849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 08:57:34.456604 (+ 1755us) service_pool.cc:225] Handling call
0112 08:57:35.106865 (+650261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 08:57:35.113219 23715 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Refusing update from remote peer e7c2dbf455f04f4fbcc282c86eae73e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 08:57:35.119215 23831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:35.186403 23876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:35.187355 23876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:35.188134 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:35.189109 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:35.264688 23898 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 08:57:35.300602 23898 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:35.300998 23898 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: loading cluster ID for follower catalog manager: success
I20250112 08:57:35.308615 23898 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: acquiring CA information for follower catalog manager: success
I20250112 08:57:35.325544 23898 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:35.624382 23885 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:35.624873 23885 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:35.629575 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:35.630049 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:35.631721 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:35.627110 23885 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:35.632591 23876 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:35.654330 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:35.654996 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:35.655966 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 08:57:35.678004 23831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:35.678932 23831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
W20250112 08:57:35.732134 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:35.744417 23875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.193
I20250112 08:57:35.924643 23875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.42.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.193
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:35.926472 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:35.928660 23875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:35.997009 23907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:35.997176 23908 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:36.000686 23908 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:36.010589 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:36.013624 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:36.016297 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:36.019038 23908 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:36.020027 23908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:36.181659 23915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:36.182415 23915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:36.186209 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:36.187741 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:36.190904 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:36.195079 23915 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:36.195751 23915 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:36.777312 23917 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:36.777791 23917 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:36.780540 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:36.797442 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:36.798988 23694 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:36.804418 23917 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:36.808239 23917 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:37.007757 23919 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:37.008602 23919 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:37.011293 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:37.017295 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:37.018895 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:37.020743 23919 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:37.021588 23919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:38.204999 23910 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2193 milliseconds
W20250112 08:57:38.202462 23875 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 2.194s user 0.245s sys 0.617s
W20250112 08:57:38.204712 23912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:38.205752 23875 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 2.197s user 0.246s sys 0.617s
W20250112 08:57:37.367307 23906 debug-util.cc:398] Leaking SignalData structure 0x7b080001c400 after lost signal to thread 23875
I20250112 08:57:38.206126 23875 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250112 08:57:36.012557 23909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:38.210829 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:38.213635 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:38.215142 23875 hybrid_clock.cc:648] HybridClock initialized: now 1736672258215073 us; error 79 us; skew 500 ppm
I20250112 08:57:38.216274 23875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:38.230368 23875 webserver.cc:458] Webserver started at http://127.21.42.193:39071/ using document root <none> and password file <none>
I20250112 08:57:38.231649 23875 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:38.231962 23875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:38.232542 23875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:38.239992 23875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data/instance:
uuid: "6ff253b3e7a14f43b118105fda65322b"
format_stamp: "Formatted at 2025-01-12 08:57:38 on dist-test-slave-ghv6"
I20250112 08:57:38.241528 23875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal/instance:
uuid: "6ff253b3e7a14f43b118105fda65322b"
format_stamp: "Formatted at 2025-01-12 08:57:38 on dist-test-slave-ghv6"
I20250112 08:57:38.251384 23875 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20250112 08:57:38.259109 23925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:38.260478 23875 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250112 08:57:38.260869 23875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal
uuid: "6ff253b3e7a14f43b118105fda65322b"
format_stamp: "Formatted at 2025-01-12 08:57:38 on dist-test-slave-ghv6"
I20250112 08:57:38.261358 23875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:38.346900 23875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:38.348927 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:38.349548 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:38.353870 23875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:57:38.359818 23875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:57:38.360147 23875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250112 08:57:38.360472 23875 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:57:38.360684 23875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:38.560122 24014 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:38.566308 24014 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:38.569619 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:38.571760 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:38.573843 23694 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:38.579594 24014 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:38.580704 24014 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:38.646090 23875 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.193:42101
I20250112 08:57:38.646205 24040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.193:42101 every 8 connection(s)
I20250112 08:57:38.649152 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/data/info.pb
I20250112 08:57:38.657655 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 23875
I20250112 08:57:38.658542 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal/instance
I20250112 08:57:38.686491 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.194:0
--local_ip_for_outbound_sockets=127.21.42.194
--webserver_interface=127.21.42.194
--webserver_port=0
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:57:38.765461 24041 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:38.766083 24041 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:38.769381 24041 heartbeater.cc:510] Master 127.21.42.252:36643 requested a full tablet report, sending...
I20250112 08:57:38.772974 23785 ts_manager.cc:194] Registered new tserver with Master: 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:38.775715 23785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.193:47405
I20250112 08:57:38.789125 24042 heartbeater.cc:346] Connected to a master server at 127.21.42.253:36201
I20250112 08:57:38.789491 24042 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:38.790165 24042 heartbeater.cc:510] Master 127.21.42.253:36201 requested a full tablet report, sending...
I20250112 08:57:38.793452 23707 ts_manager.cc:194] Registered new tserver with Master: 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:38.802300 24045 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:38.802675 24045 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:38.803377 24045 heartbeater.cc:510] Master 127.21.42.254:41739 requested a full tablet report, sending...
I20250112 08:57:38.813127 23633 ts_manager.cc:194] Registered new tserver with Master: 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:38.866398 24052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:38.867156 24052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:38.870115 24052 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:38.871249 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:38.872555 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:38.874588 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:38.875628 24052 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:39.086331 24016 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:39.086833 24016 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:39.088820 24016 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:39.089774 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:39.090219 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:39.091678 23694 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:39.092391 24016 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:39.150066 24048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:39.150589 24048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:39.151175 24048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:39.185348 24048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:39.186625 24048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.194
I20250112 08:57:39.224978 24048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.42.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.194
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:39.226325 24048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:39.228188 24048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:39.248155 24058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:39.282764 24041 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:39.457387 24065 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:39.458149 24065 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:39.460040 24065 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:39.461104 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:39.461391 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:39.463420 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:39.464689 24065 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:40.084025 24066 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:40.084424 24066 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:40.086190 24066 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:40.087344 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:40.087672 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:40.089133 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a6dc008c89249efbae18aa4e34b9ea8; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:40.089828 24066 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:40.435048 24067 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:40.435573 24067 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:40.437952 24067 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:40.438313 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:40.438977 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:40.440199 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:40.440757 24067 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:39.248800 24059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:39.250171 24061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:39.251096 24048 server_base.cc:1034] running on GCE node
I20250112 08:57:40.489275 24048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:40.491446 24048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:40.492781 24048 hybrid_clock.cc:648] HybridClock initialized: now 1736672260492764 us; error 50 us; skew 500 ppm
I20250112 08:57:40.493570 24048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:40.500241 24048 webserver.cc:458] Webserver started at http://127.21.42.194:42989/ using document root <none> and password file <none>
I20250112 08:57:40.501152 24048 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:40.501358 24048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:40.501777 24048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:40.506642 24048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data/instance:
uuid: "8f4921885fee48319c7132725c7691bf"
format_stamp: "Formatted at 2025-01-12 08:57:40 on dist-test-slave-ghv6"
I20250112 08:57:40.507745 24048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal/instance:
uuid: "8f4921885fee48319c7132725c7691bf"
format_stamp: "Formatted at 2025-01-12 08:57:40 on dist-test-slave-ghv6"
I20250112 08:57:40.514818 24048 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250112 08:57:40.521517 24072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:40.522580 24048 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250112 08:57:40.522903 24048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal
uuid: "8f4921885fee48319c7132725c7691bf"
format_stamp: "Formatted at 2025-01-12 08:57:40 on dist-test-slave-ghv6"
I20250112 08:57:40.523208 24048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:40.580775 24048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:40.582240 24048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:40.582643 24048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:40.585407 24048 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:57:40.589677 24048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:57:40.589898 24048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:40.590101 24048 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:57:40.590243 24048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:40.734226 24048 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.194:37375
I20250112 08:57:40.734336 24184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.194:37375 every 8 connection(s)
I20250112 08:57:40.736811 24048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/data/info.pb
I20250112 08:57:40.738826 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 24048
I20250112 08:57:40.739249 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal/instance
I20250112 08:57:40.755331 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.42.195:0
--local_ip_for_outbound_sockets=127.21.42.195
--webserver_interface=127.21.42.195
--webserver_port=0
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 08:57:40.785157 24185 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:40.785768 24185 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:40.786940 24185 heartbeater.cc:510] Master 127.21.42.252:36643 requested a full tablet report, sending...
I20250112 08:57:40.790079 23785 ts_manager.cc:194] Registered new tserver with Master: 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:40.790326 24187 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:40.790656 24187 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:40.791533 24187 heartbeater.cc:510] Master 127.21.42.254:41739 requested a full tablet report, sending...
I20250112 08:57:40.791730 23785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.194:50369
I20250112 08:57:40.794167 23633 ts_manager.cc:194] Registered new tserver with Master: 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:40.794414 24186 heartbeater.cc:346] Connected to a master server at 127.21.42.253:36201
I20250112 08:57:40.794788 24186 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:40.795636 24186 heartbeater.cc:510] Master 127.21.42.253:36201 requested a full tablet report, sending...
I20250112 08:57:40.799508 23707 ts_manager.cc:194] Registered new tserver with Master: 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:41.020135 24196 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:41.020524 24196 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:41.022902 24196 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:41.023108 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:41.023571 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:41.024797 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:41.025368 24196 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 08:57:41.094517 24195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:41.094980 24195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:41.095506 24195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:41.127702 24195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:41.128633 24195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.42.195
I20250112 08:57:41.168823 24195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.42.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.42.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.42.195
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:41.170522 24195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:41.172274 24195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:41.189945 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:41.297534 24185 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:41.532641 24209 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:41.533108 24209 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:41.535056 24209 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:41.536317 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:41.536877 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:41.538933 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:41.539878 24209 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:42.059093 24210 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:42.059533 24210 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.061179 24210 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:42.062222 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:42.062306 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:42.063975 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:42.064550 24210 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 08:57:42.075028 24211 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:42.075501 24211 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.077293 24211 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:42.078347 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:42.078617 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:42.079222 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a6dc008c89249efbae18aa4e34b9ea8 in term 1.
I20250112 08:57:42.080399 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8; no voters: e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:42.081205 24211 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 08:57:42.081534 24211 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:42.081852 24211 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 1 FOLLOWER]: Advancing to term 2
I20250112 08:57:42.088024 24211 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.089954 24211 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 election: Requested vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:42.090425 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
I20250112 08:57:42.090973 23643 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 1 FOLLOWER]: Advancing to term 2
I20250112 08:57:42.091205 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
I20250112 08:57:42.097390 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a6dc008c89249efbae18aa4e34b9ea8 in term 2.
I20250112 08:57:42.098452 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8; no voters: e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:42.099174 24211 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 08:57:42.100348 24211 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 LEADER]: Becoming Leader. State: Replica: 5a6dc008c89249efbae18aa4e34b9ea8, State: Running, Role: LEADER
I20250112 08:57:42.101234 24211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.104979 23715 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 LEADER]: Rejecting Update request from peer e7c2dbf455f04f4fbcc282c86eae73e1 for earlier term 1. Current term is 2. Ops: []
I20250112 08:57:42.105736 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6dc008c89249efbae18aa4e34b9ea8. Latest consensus state: current_term: 2 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:42.106977 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:42.108263 24212 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250112 08:57:42.108986 24215 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:57:42.110774 24212 consensus_queue.cc:1225] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Peer 5a6dc008c89249efbae18aa4e34b9ea8 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250112 08:57:42.111451 24216 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 08:57:42.112180 24216 raft_consensus.cc:738] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Running, Role: LEADER
I20250112 08:57:42.113432 24216 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.115336 24216 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 1 FOLLOWER]: Advancing to term 2
I20250112 08:57:42.115859 24215 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:57:42.118956 24215 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:42.119165 24215 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:57:42.122648 24215 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:42.125586 24215 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Loaded TSK: 0
I20250112 08:57:42.126912 24215 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:42.130897 24215 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:42.131484 23864 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:42.133844 24215 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
W20250112 08:57:41.190722 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:41.191201 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:41.192567 24195 server_base.cc:1034] running on GCE node
I20250112 08:57:42.425658 24195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:42.428149 24195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:42.429493 24195 hybrid_clock.cc:648] HybridClock initialized: now 1736672262429458 us; error 56 us; skew 500 ppm
I20250112 08:57:42.430305 24195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:42.436901 24195 webserver.cc:458] Webserver started at http://127.21.42.195:39295/ using document root <none> and password file <none>
I20250112 08:57:42.437912 24195 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:42.438105 24195 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:42.438508 24195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 08:57:42.443606 24195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data/instance:
uuid: "7679613902d342678beaffab898dfd7d"
format_stamp: "Formatted at 2025-01-12 08:57:42 on dist-test-slave-ghv6"
I20250112 08:57:42.444829 24195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal/instance:
uuid: "7679613902d342678beaffab898dfd7d"
format_stamp: "Formatted at 2025-01-12 08:57:42 on dist-test-slave-ghv6"
I20250112 08:57:42.451768 24195 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250112 08:57:42.458001 24221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:42.459179 24195 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250112 08:57:42.459501 24195 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal
uuid: "7679613902d342678beaffab898dfd7d"
format_stamp: "Formatted at 2025-01-12 08:57:42 on dist-test-slave-ghv6"
I20250112 08:57:42.459847 24195 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:42.507481 24216 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:42.507858 24216 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.509497 24216 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
I20250112 08:57:42.510331 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:42.510705 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" is_pre_election: true
I20250112 08:57:42.510888 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 2.
I20250112 08:57:42.511778 23769 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters:
I20250112 08:57:42.512701 24216 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 08:57:42.512919 24216 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:42.513163 24216 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 2 FOLLOWER]: Advancing to term 3
I20250112 08:57:42.517622 24216 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.519059 24216 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 3 election: Requested vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
I20250112 08:57:42.519730 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
I20250112 08:57:42.520130 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a6dc008c89249efbae18aa4e34b9ea8"
I20250112 08:57:42.520342 23643 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 2 FOLLOWER]: Advancing to term 3
I20250112 08:57:42.524987 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 3.
I20250112 08:57:42.525782 23769 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:42.526475 24216 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 08:57:42.526819 24216 raft_consensus.cc:695] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 LEADER]: Becoming Leader. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Running, Role: LEADER
I20250112 08:57:42.527419 24216 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.530916 24212 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 3 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:42.531646 24212 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:42.540779 24227 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:57:42.547392 24227 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:57:42.548251 23643 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 3 FOLLOWER]: Rejecting Update request from peer 5a6dc008c89249efbae18aa4e34b9ea8 for earlier term 2. Current term is 3. Ops: []
I20250112 08:57:42.549245 24195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:42.549700 24213 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 08:57:42.550580 24213 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 08:57:42.550858 24213 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a6dc008c89249efbae18aa4e34b9ea8, State: Running, Role: LEADER
I20250112 08:57:42.551100 24195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:42.551489 24227 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:42.551620 24195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:42.551728 24227 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:57:42.551529 24213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:42.552994 24213 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 2 FOLLOWER]: Advancing to term 3
I20250112 08:57:42.557389 24227 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:42.561699 24227 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Loaded TSK: 0
I20250112 08:57:42.562891 24227 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:42.563364 24195 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 08:57:42.566109 24227 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:42.569052 24195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 08:57:42.569329 24195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:42.569459 24227 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 08:57:42.569613 24195 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 08:57:42.569813 24195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:42.718194 24195 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.195:41233
I20250112 08:57:42.718318 24335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.195:41233 every 8 connection(s)
I20250112 08:57:42.720836 24195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/data/info.pb
I20250112 08:57:42.731319 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 24195
I20250112 08:57:42.731968 21675 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal/instance
I20250112 08:57:42.772483 24338 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:42.773020 24338 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:42.776755 24338 heartbeater.cc:510] Master 127.21.42.254:41739 requested a full tablet report, sending...
I20250112 08:57:42.778896 24336 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:42.779362 24336 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:42.780279 24336 heartbeater.cc:510] Master 127.21.42.252:36643 requested a full tablet report, sending...
I20250112 08:57:42.781339 23633 ts_manager.cc:194] Registered new tserver with Master: 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:42.783346 24337 heartbeater.cc:346] Connected to a master server at 127.21.42.253:36201
I20250112 08:57:42.783681 24337 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:42.784585 24337 heartbeater.cc:510] Master 127.21.42.253:36201 requested a full tablet report, sending...
I20250112 08:57:42.784787 23785 ts_manager.cc:194] Registered new tserver with Master: 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:42.787206 23707 ts_manager.cc:194] Registered new tserver with Master: 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:42.787279 23785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.42.195:45693
I20250112 08:57:42.804615 21675 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 08:57:42.813637 24186 heartbeater.cc:502] Master 127.21.42.253:36201 was elected leader, sending a full tablet report...
I20250112 08:57:42.825202 24042 heartbeater.cc:502] Master 127.21.42.253:36201 was elected leader, sending a full tablet report...
I20250112 08:57:42.876426 23784 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35162:
name: "default.Table_d16629577a884abab7f9b83a12d4bc97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:42.879340 23785 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35162:
name: "default.Table_64def52107a64d539a649b582e39ae08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:42.909729 23783 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35162:
name: "default.Table_7c040710b44a4de7bb1a466c66def679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:42.912272 23782 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35162:
name: "default.Table_d4842570453f4558bc5d55fc12813c0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:42.914757 23784 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d16629577a884abab7f9b83a12d4bc97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:42.929337 23785 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_64def52107a64d539a649b582e39ae08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:42.933842 23782 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d4842570453f4558bc5d55fc12813c0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:42.959950 21675 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu with pid 23743
W20250112 08:57:42.990060 24349 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.42.252:36643)
W20250112 08:57:43.033221 24349 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.21.42.252:36643, user_credentials={real_user=slave}} blocked reactor thread for 47002.8us
I20250112 08:57:43.060215 24213 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:43.060683 24213 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:43.062566 24213 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:43.063297 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:43.063879 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a6dc008c89249efbae18aa4e34b9ea8 in term 3.
I20250112 08:57:43.064994 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8; no voters:
I20250112 08:57:43.065761 24213 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250112 08:57:43.066071 24213 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:43.066383 24213 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 3 FOLLOWER]: Advancing to term 4
I20250112 08:57:43.073423 24213 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:43.077138 24213 leader_election.cc:290] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 election: Requested vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:43.077122 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
I20250112 08:57:43.077812 23643 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 3 FOLLOWER]: Advancing to term 4
W20250112 08:57:43.080592 23694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250112 08:57:43.086640 23643 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a6dc008c89249efbae18aa4e34b9ea8 in term 4.
W20250112 08:57:43.092483 23694 leader_election.cc:336] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:43.094285 23691 leader_election.cc:304] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8; no voters: e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:43.095345 24213 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 4 FOLLOWER]: Leader election won for term 4
W20250112 08:57:43.099505 23694 leader_election.cc:336] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:43.100486 24213 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [term 4 LEADER]: Becoming Leader. State: Replica: 5a6dc008c89249efbae18aa4e34b9ea8, State: Running, Role: LEADER
I20250112 08:57:43.101325 24213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:43.107142 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6dc008c89249efbae18aa4e34b9ea8. Latest consensus state: current_term: 4 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:43.108139 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:43.109822 24374 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:57:43.115576 24374 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:57:43.119532 24374 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:43.119794 24374 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:57:43.125155 24374 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:43.129567 24374 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Loaded TSK: 0
I20250112 08:57:43.130573 24374 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:43.134850 24374 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:43.139124 24374 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 08:57:43.156935 23704 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_d16629577a884abab7f9b83a12d4bc97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.157059 23706 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_7c040710b44a4de7bb1a466c66def679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.158020 23705 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_64def52107a64d539a649b582e39ae08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.155318 23707 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_d4842570453f4558bc5d55fc12813c0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 08:57:43.194831 23704 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d16629577a884abab7f9b83a12d4bc97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:43.202097 23706 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7c040710b44a4de7bb1a466c66def679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:43.202072 23705 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_64def52107a64d539a649b582e39ae08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 08:57:43.208700 23707 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d4842570453f4558bc5d55fc12813c0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 08:57:43.293361 24336 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
W20250112 08:57:43.298381 24336 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:43.313387 24185 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:43.315447 24041 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
W20250112 08:57:43.317739 24185 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:43.321107 24041 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:43.513994 23706 catalog_manager.cc:2149] created HMS catalog entry for table [id=d8ab19d75e784220b1c16aca0d90678c]
W20250112 08:57:43.522917 23694 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 -> Peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Couldn't send request to peer e7c2dbf455f04f4fbcc282c86eae73e1. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:43.526790 23643 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Refusing update from remote peer 5a6dc008c89249efbae18aa4e34b9ea8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250112 08:57:43.529377 24213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 08:57:43.549124 24379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6dc008c89249efbae18aa4e34b9ea8. Latest consensus state: current_term: 4 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:43.549898 24379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:43.555150 24378 mvcc.cc:204] Tried to move back new op lower bound from 7113409591393685504 to 7113409585574133760. Current Snapshot: MvccSnapshot[applied={T|T < 7113409591393685504}]
I20250112 08:57:43.581043 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:43.581815 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:43.585297 23706 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "default.Table_7c040710b44a4de7bb1a466c66def679" } new_table_name: "default.Table_34ded4956e99401f95167457615dd003" modify_external_catalogs: true
I20250112 08:57:43.597268 23705 catalog_manager.cc:2149] created HMS catalog entry for table [id=0302d2aee6434879a8f4e7fe405748c2]
I20250112 08:57:43.613574 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:43.614460 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:43.617743 24379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:43.618407 24379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:43.701998 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 29b55054356041659fd95439f384584f (DEFAULT_TABLE table=default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.704761 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b55054356041659fd95439f384584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.710152 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:36643
--webserver_interface=127.21.42.252
--webserver_port=38649
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--hive_metastore_uris=thrift://127.0.0.1:39997
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 08:57:43.737507 24271 tablet_service.cc:1467] Processing CreateTablet for tablet 29b55054356041659fd95439f384584f (DEFAULT_TABLE table=default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.741427 23974 tablet_service.cc:1467] Processing CreateTablet for tablet 29b55054356041659fd95439f384584f (DEFAULT_TABLE table=default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.743530 23974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b55054356041659fd95439f384584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.744793 24271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29b55054356041659fd95439f384584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.754057 23705 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "default.Table_64def52107a64d539a649b582e39ae08" } new_table_name: "default.Table_826bc0c69bef4c9981e5942d7127fb86" modify_external_catalogs: true
I20250112 08:57:43.777439 23973 tablet_service.cc:1467] Processing CreateTablet for tablet 77a9b2f822354182b0e36c55cce850ce (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.778649 23973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9b2f822354182b0e36c55cce850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.778932 24119 tablet_service.cc:1467] Processing CreateTablet for tablet 77a9b2f822354182b0e36c55cce850ce (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.779994 24119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9b2f822354182b0e36c55cce850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.781411 24270 tablet_service.cc:1467] Processing CreateTablet for tablet 77a9b2f822354182b0e36c55cce850ce (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:43.782505 24270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9b2f822354182b0e36c55cce850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:43.787791 23703 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_8b0ef96b104f4089b1023146b02467bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.797279 24337 heartbeater.cc:502] Master 127.21.42.253:36201 was elected leader, sending a full tablet report...
I20250112 08:57:43.821550 24186 heartbeater.cc:502] Master 127.21.42.253:36201 was elected leader, sending a full tablet report...
I20250112 08:57:43.832199 24042 heartbeater.cc:502] Master 127.21.42.253:36201 was elected leader, sending a full tablet report...
I20250112 08:57:43.834224 24393 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:43.842655 24393 tablet_bootstrap.cc:654] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:43.850476 23701 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_49e783a6610044bc9fc98032934075db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.856086 24393 log.cc:826] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:43.881011 24393 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:43.881635 24393 ts_tablet_manager.cc:1397] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.014s
I20250112 08:57:43.898638 24398 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
I20250112 08:57:43.915704 24393 raft_consensus.cc:357] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:43.916800 24393 raft_consensus.cc:383] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:43.917173 24393 raft_consensus.cc:738] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
I20250112 08:57:43.918147 24398 tablet_bootstrap.cc:654] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:43.918206 24393 consensus_queue.cc:260] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:43.922999 24398 log.cc:826] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:43.930150 23702 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35266
08:57:43.931 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_d4842570453f4558bc5d55fc12813c0e already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250112 08:57:43.935166 23707 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=19d55c36f679497ca273d25bcc980604]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_d4842570453f4558bc5d55fc12813c0e already exists)
I20250112 08:57:43.938860 24398 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:43.939508 24398 ts_tablet_manager.cc:1397] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.009s
I20250112 08:57:43.944891 23700 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35266
I20250112 08:57:43.945912 23699 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_973a1a0f49a3470788ec010b2b1ceaa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:43.953568 24393 ts_tablet_manager.cc:1428] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.072s user 0.029s sys 0.015s
I20250112 08:57:43.956054 24393 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:43.959250 24399 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
I20250112 08:57:43.963382 24393 tablet_bootstrap.cc:654] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:43.984649 23702 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Sending DeleteTablet for 3 replicas of tablet 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:43.985071 24393 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:43.985633 24393 ts_tablet_manager.cc:1397] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.001s
I20250112 08:57:43.988272 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a9b2f822354182b0e36c55cce850ce with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfa21f18e8f4c8da4681ec1ef221720) from {username='slave'} at 127.0.0.1:44620
I20250112 08:57:43.988451 24393 raft_consensus.cc:357] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:43.989292 23702 catalog_manager.cc:6596] ReplaceTablet: tablet 77a9b2f822354182b0e36c55cce850ce deleted and replaced by tablet 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:43.989395 24393 raft_consensus.cc:383] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:43.989876 24393 raft_consensus.cc:738] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
I20250112 08:57:43.990958 24393 consensus_queue.cc:260] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:43.995772 24393 ts_tablet_manager.cc:1428] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250112 08:57:43.997462 24403 tablet_replica.cc:331] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: stopping tablet replica
I20250112 08:57:43.998394 24403 raft_consensus.cc:2238] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:43.999028 24403 raft_consensus.cc:2267] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:44.002418 24403 ts_tablet_manager.cc:1905] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.013254 23974 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a9b2f822354182b0e36c55cce850ce with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfa21f18e8f4c8da4681ec1ef221720) from {username='slave'} at 127.0.0.1:50164
I20250112 08:57:44.016099 23973 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.016824 24270 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.017243 23973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfa21f18e8f4c8da4681ec1ef221720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.018177 24270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfa21f18e8f4c8da4681ec1ef221720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.018543 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 08:57:44.019138 24194 tablet.cc:2367] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:57:44.019749 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dfa21f18e8f4c8da4681ec1ef221720. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.022488 24271 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a9b2f822354182b0e36c55cce850ce with delete_type TABLET_DATA_DELETED (replaced by tablet 8dfa21f18e8f4c8da4681ec1ef221720) from {username='slave'} at 127.0.0.1:33160
I20250112 08:57:44.027272 24403 ts_tablet_manager.cc:1918] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:44.027788 24403 log.cc:1198] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.028786 24403 ts_tablet_manager.cc:1939] T 77a9b2f822354182b0e36c55cce850ce P 8f4921885fee48319c7132725c7691bf: Deleting consensus metadata
I20250112 08:57:44.031975 23691 catalog_manager.cc:4872] TS 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): tablet 77a9b2f822354182b0e36c55cce850ce (table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]) successfully deleted
W20250112 08:57:44.034857 23693 catalog_manager.cc:4852] TS 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): delete failed for tablet 77a9b2f822354182b0e36c55cce850ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77a9b2f822354182b0e36c55cce850ce already in progress: creating tablet
I20250112 08:57:44.035084 24398 raft_consensus.cc:357] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:44.036049 24398 raft_consensus.cc:383] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.036330 24398 raft_consensus.cc:738] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.039229 24398 consensus_queue.cc:260] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
W20250112 08:57:44.043239 23691 catalog_manager.cc:4852] TS 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): delete failed for tablet 77a9b2f822354182b0e36c55cce850ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77a9b2f822354182b0e36c55cce850ce already in progress: creating tablet
I20250112 08:57:44.047559 24393 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:44.054598 24393 tablet_bootstrap.cc:654] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.063968 24398 ts_tablet_manager.cc:1428] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.124s user 0.035s sys 0.005s
I20250112 08:57:44.064967 24398 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
W20250112 08:57:44.066237 23702 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.068444 24393 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:44.068969 24393 ts_tablet_manager.cc:1397] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.006s
I20250112 08:57:44.071303 24398 tablet_bootstrap.cc:654] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.072160 24393 raft_consensus.cc:357] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.073009 24393 raft_consensus.cc:383] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.073364 24393 raft_consensus.cc:738] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.074185 24393 consensus_queue.cc:260] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.076879 24393 ts_tablet_manager.cc:1428] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250112 08:57:44.088315 23704 catalog_manager.cc:2149] created HMS catalog entry for table [id=ad737da0fbd64899bf4d28c3199895ca]
F20250112 08:57:44.095674 23896 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_d4842570453f4558bc5d55fc12813c0e
*** Check failure stack trace: ***
I20250112 08:57:44.112933 23704 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
I20250112 08:57:44.113679 23704 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35266:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
@ 0x7f83aa628950 google::LogMessage::Flush() at ??:0
W20250112 08:57:44.124836 23702 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
@ 0x7f83aa6286bf google::LogMessage::~LogMessage() at ??:0
I20250112 08:57:44.146950 24271 tablet_service.cc:1467] Processing CreateTablet for tablet 8677901d662049c6b6fa75145ca455b1 (DEFAULT_TABLE table=default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.148161 24271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8677901d662049c6b6fa75145ca455b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.047848 24399 tablet_bootstrap.cc:654] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.158335 23973 tablet_service.cc:1467] Processing CreateTablet for tablet 8677901d662049c6b6fa75145ca455b1 (DEFAULT_TABLE table=default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.158803 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 8677901d662049c6b6fa75145ca455b1 (DEFAULT_TABLE table=default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.159480 23973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8677901d662049c6b6fa75145ca455b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.159920 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8677901d662049c6b6fa75145ca455b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.161303 24399 log.cc:826] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:44.163647 24400 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.164203 24400 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.176841 24393 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:44.176851 23707 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35266
I20250112 08:57:44.176352 24400 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:44.183298 24393 tablet_bootstrap.cc:654] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.185864 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfa21f18e8f4c8da4681ec1ef221720" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
W20250112 08:57:44.188095 24075 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.199678 24393 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:44.199952 23707 master_service.cc:946] ReplaceTablet: received request to replace tablet 8dfa21f18e8f4c8da4681ec1ef221720 from {username='slave'} at 127.0.0.1:35266
I20250112 08:57:44.200266 24393 ts_tablet_manager.cc:1397] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.014s
I20250112 08:57:44.204370 24393 raft_consensus.cc:357] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.206604 24393 raft_consensus.cc:383] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.207388 24393 raft_consensus.cc:738] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.208748 24393 consensus_queue.cc:260] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
@ 0x7f83b552382a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250112 08:57:44.211977 24393 ts_tablet_manager.cc:1428] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
@ 0x7f83b552307c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250112 08:57:44.249645 23707 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Sending DeleteTablet for 3 replicas of tablet 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:44.251164 23707 catalog_manager.cc:6596] ReplaceTablet: tablet 8dfa21f18e8f4c8da4681ec1ef221720 deleted and replaced by tablet b4277264769840449591e82ca21bec48
I20250112 08:57:44.251448 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 with delete_type TABLET_DATA_DELETED (replaced by tablet b4277264769840449591e82ca21bec48) from {username='slave'} at 127.0.0.1:50164
I20250112 08:57:44.253033 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 with delete_type TABLET_DATA_DELETED (replaced by tablet b4277264769840449591e82ca21bec48) from {username='slave'} at 127.0.0.1:44620
W20250112 08:57:44.253046 23693 catalog_manager.cc:4852] TS 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): delete failed for tablet 8dfa21f18e8f4c8da4681ec1ef221720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dfa21f18e8f4c8da4681ec1ef221720 already in progress: creating tablet
I20250112 08:57:44.253782 24403 tablet_replica.cc:331] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: stopping tablet replica
I20250112 08:57:44.254732 24403 raft_consensus.cc:2238] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:44.255308 24403 raft_consensus.cc:2267] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7f83b5526d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250112 08:57:44.259238 24403 ts_tablet_manager.cc:1905] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.276242 24270 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 with delete_type TABLET_DATA_DELETED (replaced by tablet b4277264769840449591e82ca21bec48) from {username='slave'} at 127.0.0.1:33160
W20250112 08:57:44.277984 23691 catalog_manager.cc:4852] TS 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): delete failed for tablet 8dfa21f18e8f4c8da4681ec1ef221720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dfa21f18e8f4c8da4681ec1ef221720 already in progress: creating tablet
W20250112 08:57:44.281351 24051 tablet.cc:2367] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7f83b5526cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250112 08:57:44.284011 24399 tablet_bootstrap.cc:492] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:44.284536 24399 ts_tablet_manager.cc:1397] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.327s user 0.013s sys 0.003s
I20250112 08:57:44.286419 24398 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:44.286829 24398 ts_tablet_manager.cc:1397] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.222s user 0.027s sys 0.121s
I20250112 08:57:44.290063 24398 raft_consensus.cc:357] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.290858 24398 raft_consensus.cc:383] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.291157 24398 raft_consensus.cc:738] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.218492 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dfa21f18e8f4c8da4681ec1ef221720" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
I20250112 08:57:44.296350 24403 ts_tablet_manager.cc:1918] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:44.296782 24403 log.cc:1198] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:44.297482 24403 ts_tablet_manager.cc:1939] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: Deleting consensus metadata
I20250112 08:57:44.299286 23691 catalog_manager.cc:4872] TS 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): tablet 8dfa21f18e8f4c8da4681ec1ef221720 (table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]) successfully deleted
@ 0x7f83b5526c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250112 08:57:44.313936 23700 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35266:
name: "default.Table_a08e8ce40f9a4a72bfd0c37e4d8b5c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:44.315946 23973 tablet_service.cc:1467] Processing CreateTablet for tablet b4277264769840449591e82ca21bec48 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.316295 24120 tablet_service.cc:1467] Processing CreateTablet for tablet b4277264769840449591e82ca21bec48 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.316991 23973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4277264769840449591e82ca21bec48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.317306 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4277264769840449591e82ca21bec48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.325677 24271 tablet_service.cc:1467] Processing CreateTablet for tablet b4277264769840449591e82ca21bec48 (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:44.326737 24271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4277264769840449591e82ca21bec48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:44.328120 24398 consensus_queue.cc:260] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.330291 24398 ts_tablet_manager.cc:1428] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
W20250112 08:57:44.330358 24073 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.330806 24073 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f4921885fee48319c7132725c7691bf; no voters: 6ff253b3e7a14f43b118105fda65322b, 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.331171 24398 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
W20250112 08:57:44.331538 24073 raft_consensus.cc:2681] T 8dfa21f18e8f4c8da4681ec1ef221720 P 8f4921885fee48319c7132725c7691bf: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 08:57:44.349538 24398 tablet_bootstrap.cc:654] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
@ 0x7f83b5526c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250112 08:57:44.359061 24393 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:44.365290 24393 tablet_bootstrap.cc:654] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.380913 24393 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:44.381364 24393 ts_tablet_manager.cc:1397] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.001s
I20250112 08:57:44.384022 24393 raft_consensus.cc:357] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.384793 24393 raft_consensus.cc:383] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.385085 24393 raft_consensus.cc:738] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
@ 0x7f83b5525f3e std::__1::__function::__func<>::operator()() at ??:0
I20250112 08:57:44.385713 23707 master_service.cc:946] ReplaceTablet: received request to replace tablet 8dfa21f18e8f4c8da4681ec1ef221720 from {username='slave'} at 127.0.0.1:35266
I20250112 08:57:44.375456 24399 raft_consensus.cc:357] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
I20250112 08:57:44.386732 24399 raft_consensus.cc:383] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.387116 24399 raft_consensus.cc:738] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250112 08:57:44.389619 24399 consensus_queue.cc:260] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:44.390622 24398 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:44.391088 24398 ts_tablet_manager.cc:1397] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.004s
I20250112 08:57:44.392277 24393 consensus_queue.cc:260] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.393942 24398 raft_consensus.cc:357] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.394755 24398 raft_consensus.cc:383] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.395048 24398 raft_consensus.cc:738] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.397826 24393 ts_tablet_manager.cc:1428] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250112 08:57:44.400158 24399 ts_tablet_manager.cc:1428] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.115s user 0.022s sys 0.006s
I20250112 08:57:44.401038 24399 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
W20250112 08:57:44.402316 23702 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.395669 24398 consensus_queue.cc:260] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.409595 24399 tablet_bootstrap.cc:654] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
W20250112 08:57:44.412407 23698 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.416154 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 with delete_type TABLET_DATA_DELETED (replaced by tablet b4277264769840449591e82ca21bec48) from {username='slave'} at 127.0.0.1:50164
W20250112 08:57:44.417805 23693 catalog_manager.cc:4852] TS 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): delete failed for tablet 8dfa21f18e8f4c8da4681ec1ef221720 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dfa21f18e8f4c8da4681ec1ef221720 already in progress: creating tablet
I20250112 08:57:44.419272 24398 ts_tablet_manager.cc:1428] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
@ 0x7f83ac86d8d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250112 08:57:44.430018 24398 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
I20250112 08:57:44.430531 24400 raft_consensus.cc:491] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.430979 24400 raft_consensus.cc:513] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.432883 24400 leader_election.cc:290] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:44.433354 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dfa21f18e8f4c8da4681ec1ef221720 with delete_type TABLET_DATA_DELETED (replaced by tablet b4277264769840449591e82ca21bec48) from {username='slave'} at 127.0.0.1:50164
I20250112 08:57:44.434432 24404 tablet_replica.cc:331] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: stopping tablet replica
I20250112 08:57:44.434677 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b55054356041659fd95439f384584f" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
I20250112 08:57:44.435050 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b55054356041659fd95439f384584f" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:44.435793 23994 raft_consensus.cc:2463] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4921885fee48319c7132725c7691bf in term 0.
W20250112 08:57:44.437502 24073 leader_election.cc:343] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.435245 24404 raft_consensus.cc:2238] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:44.439303 24404 raft_consensus.cc:2267] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:44.439245 24075 leader_election.cc:304] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters: 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.440145 24400 raft_consensus.cc:2798] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:44.440465 24400 raft_consensus.cc:491] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:44.440788 24400 raft_consensus.cc:3054] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.440869 24398 tablet_bootstrap.cc:654] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.445830 24400 raft_consensus.cc:513] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.448192 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b55054356041659fd95439f384584f" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d"
I20250112 08:57:44.448547 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b55054356041659fd95439f384584f" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b"
I20250112 08:57:44.449136 23994 raft_consensus.cc:3054] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Advancing to term 1
W20250112 08:57:44.449621 24073 leader_election.cc:343] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.450789 24399 tablet_bootstrap.cc:492] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:44.451213 24399 ts_tablet_manager.cc:1397] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.050s user 0.019s sys 0.009s
I20250112 08:57:44.452098 24398 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:44.452173 24404 ts_tablet_manager.cc:1905] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.452520 24398 ts_tablet_manager.cc:1397] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.003s
I20250112 08:57:44.454107 24399 raft_consensus.cc:357] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.454811 24399 raft_consensus.cc:383] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.455096 24399 raft_consensus.cc:738] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.455731 24399 consensus_queue.cc:260] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.468972 23994 raft_consensus.cc:2463] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4921885fee48319c7132725c7691bf in term 1.
I20250112 08:57:44.469828 24399 ts_tablet_manager.cc:1428] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250112 08:57:44.475381 23702 master_service.cc:946] ReplaceTablet: received request to replace tablet 29b55054356041659fd95439f384584f from {username='slave'} at 127.0.0.1:35266
I20250112 08:57:44.476740 24399 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
I20250112 08:57:44.476728 24075 leader_election.cc:304] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters: 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.475692 24398 raft_consensus.cc:357] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.481909 24398 raft_consensus.cc:383] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.482195 24398 raft_consensus.cc:738] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.482930 24400 leader_election.cc:290] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Requested vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:44.483181 24420 raft_consensus.cc:491] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.483700 24420 raft_consensus.cc:513] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:44.483474 24400 raft_consensus.cc:2798] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:44.485093 24400 raft_consensus.cc:695] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 LEADER]: Becoming Leader. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Running, Role: LEADER
I20250112 08:57:44.485950 24400 consensus_queue.cc:237] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } }
I20250112 08:57:44.486771 24420 leader_election.cc:290] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
@ 0x7f83adeec6db start_thread at ??:0
I20250112 08:57:44.488179 24398 consensus_queue.cc:260] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.489187 24399 tablet_bootstrap.cc:654] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.492369 24398 ts_tablet_manager.cc:1428] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250112 08:57:44.505843 24398 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
I20250112 08:57:44.505775 23702 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5a6dc008c89249efbae18aa4e34b9ea8: Sending DeleteTablet for 3 replicas of tablet 29b55054356041659fd95439f384584f
I20250112 08:57:44.507220 23702 catalog_manager.cc:6596] ReplaceTablet: tablet 29b55054356041659fd95439f384584f deleted and replaced by tablet 8c8d8e20b64b47c4a70ff492f4f11139
I20250112 08:57:44.509410 23973 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b55054356041659fd95439f384584f with delete_type TABLET_DATA_DELETED (replaced by tablet 8c8d8e20b64b47c4a70ff492f4f11139) from {username='slave'} at 127.0.0.1:50164
I20250112 08:57:44.510279 24120 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b55054356041659fd95439f384584f with delete_type TABLET_DATA_DELETED (replaced by tablet 8c8d8e20b64b47c4a70ff492f4f11139) from {username='slave'} at 127.0.0.1:44620
I20250112 08:57:44.511083 24403 tablet_replica.cc:331] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: stopping tablet replica
I20250112 08:57:44.523514 24403 raft_consensus.cc:2238] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 LEADER]: Raft consensus shutting down.
@ 0x7f83a92cd71f clone at ??:0
I20250112 08:57:44.534905 24403 pending_rounds.cc:62] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Trying to abort 1 pending ops.
I20250112 08:57:44.537308 24404 ts_tablet_manager.cc:1918] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:44.537727 24404 log.cc:1198] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:44.538941 24429 consensus_queue.cc:1159] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8f4921885fee48319c7132725c7691bf" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250112 08:57:44.539688 24400 raft_consensus.cc:491] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.540423 24400 raft_consensus.cc:513] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
W20250112 08:57:44.541153 24339 tablet.cc:2367] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 08:57:44.535205 24403 pending_rounds.cc:69] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113409595410747392 op_type: NO_OP noop_request { }
I20250112 08:57:44.543664 24403 raft_consensus.cc:2883] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 08:57:44.543843 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:44.544077 24403 raft_consensus.cc:2267] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:44.544852 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
W20250112 08:57:44.545286 24075 leader_election.cc:343] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.545822 24400 leader_election.cc:290] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:44.545846 24271 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b55054356041659fd95439f384584f with delete_type TABLET_DATA_DELETED (replaced by tablet 8c8d8e20b64b47c4a70ff492f4f11139) from {username='slave'} at 127.0.0.1:33160
I20250112 08:57:44.547653 24398 tablet_bootstrap.cc:654] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.550225 24404 ts_tablet_manager.cc:1939] T 8dfa21f18e8f4c8da4681ec1ef221720 P 6ff253b3e7a14f43b118105fda65322b: Deleting consensus metadata
I20250112 08:57:44.551066 24403 ts_tablet_manager.cc:1905] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.551381 24405 tablet_replica.cc:331] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: stopping tablet replica
I20250112 08:57:44.552238 24405 raft_consensus.cc:2238] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:44.552803 24405 raft_consensus.cc:2267] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 08:57:44.553907 24073 leader_election.cc:343] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.554347 24073 leader_election.cc:304] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f4921885fee48319c7132725c7691bf; no voters: 6ff253b3e7a14f43b118105fda65322b, 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.557345 24405 ts_tablet_manager.cc:1905] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.566174 24404 tablet_replica.cc:331] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: stopping tablet replica
I20250112 08:57:44.566936 24404 raft_consensus.cc:2238] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 08:57:44.567412 24404 raft_consensus.cc:2267] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 08:57:44.569943 24404 ts_tablet_manager.cc:1905] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 08:57:44.572095 24400 raft_consensus.cc:2743] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:44.580127 24222 leader_election.cc:343] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.586076 24398 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:44.586500 24398 ts_tablet_manager.cc:1397] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.000s
I20250112 08:57:44.589027 24398 raft_consensus.cc:357] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.589763 24398 raft_consensus.cc:383] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.590030 24398 raft_consensus.cc:738] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.590658 24398 consensus_queue.cc:260] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.594278 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a9b2f822354182b0e36c55cce850ce" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:44.594864 23994 raft_consensus.cc:2463] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7679613902d342678beaffab898dfd7d in term 0.
I20250112 08:57:44.598416 24398 ts_tablet_manager.cc:1428] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250112 08:57:44.598778 24399 tablet_bootstrap.cc:492] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:44.599668 24399 ts_tablet_manager.cc:1397] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.001s
I20250112 08:57:44.602304 24399 raft_consensus.cc:357] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.603071 24399 raft_consensus.cc:383] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.603353 24399 raft_consensus.cc:738] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.604902 24224 leader_election.cc:304] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 7679613902d342678beaffab898dfd7d; no voters: 8f4921885fee48319c7132725c7691bf
I20250112 08:57:44.606209 24420 raft_consensus.cc:2798] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:44.606539 24420 raft_consensus.cc:491] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:44.606853 24420 raft_consensus.cc:3054] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.608631 24404 ts_tablet_manager.cc:1918] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:44.609032 24404 log.cc:1198] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-0/wal/wals/29b55054356041659fd95439f384584f
I20250112 08:57:44.609900 24404 ts_tablet_manager.cc:1939] T 29b55054356041659fd95439f384584f P 6ff253b3e7a14f43b118105fda65322b: Deleting consensus metadata
I20250112 08:57:44.613781 24420 raft_consensus.cc:513] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:44.616474 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a9b2f822354182b0e36c55cce850ce" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b"
I20250112 08:57:44.617040 23994 raft_consensus.cc:3054] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.622422 24420 leader_election.cc:290] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 election: Requested vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:44.623334 23994 raft_consensus.cc:2463] T 77a9b2f822354182b0e36c55cce850ce P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7679613902d342678beaffab898dfd7d in term 1.
I20250112 08:57:44.623183 24399 consensus_queue.cc:260] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.624223 24224 leader_election.cc:304] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 7679613902d342678beaffab898dfd7d; no voters:
W20250112 08:57:44.624735 24222 leader_election.cc:343] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:44.625505 24399 ts_tablet_manager.cc:1428] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250112 08:57:44.637403 24420 raft_consensus.cc:2798] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:44.638094 24420 raft_consensus.cc:695] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [term 1 LEADER]: Becoming Leader. State: Replica: 7679613902d342678beaffab898dfd7d, State: Running, Role: LEADER
I20250112 08:57:44.639094 24420 consensus_queue.cc:237] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:44.648075 24400 raft_consensus.cc:491] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.648525 24400 raft_consensus.cc:513] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.650588 24400 leader_election.cc:290] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:44.652218 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:44.652698 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
I20250112 08:57:44.652822 23994 raft_consensus.cc:2463] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4921885fee48319c7132725c7691bf in term 0.
I20250112 08:57:44.653930 24075 leader_election.cc:304] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters:
W20250112 08:57:44.654613 24073 leader_election.cc:343] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.655360 24400 raft_consensus.cc:2798] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:44.655659 24400 raft_consensus.cc:491] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:44.655964 24400 raft_consensus.cc:3054] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.661155 24399 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
I20250112 08:57:44.662488 24400 raft_consensus.cc:513] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.621703 24405 ts_tablet_manager.cc:1918] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 08:57:44.672922 24405 log.cc:1198] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-2/wal/wals/29b55054356041659fd95439f384584f
I20250112 08:57:44.673830 24405 ts_tablet_manager.cc:1939] T 29b55054356041659fd95439f384584f P 7679613902d342678beaffab898dfd7d: Deleting consensus metadata
I20250112 08:57:44.676988 24403 ts_tablet_manager.cc:1918] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 08:57:44.677345 24403 log.cc:1198] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Deleting WAL directory at /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/ts-1/wal/wals/29b55054356041659fd95439f384584f
I20250112 08:57:44.678001 24403 ts_tablet_manager.cc:1939] T 29b55054356041659fd95439f384584f P 8f4921885fee48319c7132725c7691bf: Deleting consensus metadata
I20250112 08:57:44.679267 24406 raft_consensus.cc:491] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.679978 24406 raft_consensus.cc:513] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.680929 24400 leader_election.cc:290] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Requested vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:44.682157 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b"
I20250112 08:57:44.682672 23994 raft_consensus.cc:3054] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.696355 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "8f4921885fee48319c7132725c7691bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d"
W20250112 08:57:44.700871 24073 leader_election.cc:343] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.705524 24399 tablet_bootstrap.cc:654] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.705767 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf" is_pre_election: true
I20250112 08:57:44.706837 24140 raft_consensus.cc:2463] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 0.
I20250112 08:57:44.691596 23994 raft_consensus.cc:2463] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4921885fee48319c7132725c7691bf in term 1.
I20250112 08:57:44.698240 24406 leader_election.cc:290] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:44.708487 23927 leader_election.cc:304] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters:
I20250112 08:57:44.718961 24075 leader_election.cc:304] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters: 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.719276 24406 raft_consensus.cc:2798] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:44.719652 24406 raft_consensus.cc:491] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:44.719973 24406 raft_consensus.cc:3054] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.720026 24400 raft_consensus.cc:2798] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:44.720427 24400 raft_consensus.cc:695] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [term 1 LEADER]: Becoming Leader. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Running, Role: LEADER
I20250112 08:57:44.721151 24400 consensus_queue.cc:237] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.726764 24406 raft_consensus.cc:513] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.738440 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
W20250112 08:57:44.740082 23927 leader_election.cc:343] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.751978 24399 tablet_bootstrap.cc:492] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:44.752458 24399 ts_tablet_manager.cc:1397] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.009s
I20250112 08:57:44.752490 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d"
W20250112 08:57:44.753779 23927 leader_election.cc:343] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7679613902d342678beaffab898dfd7d (127.21.42.195:41233): Illegal state: must be running to vote when last-logged opid is not known
I20250112 08:57:44.754840 24399 raft_consensus.cc:357] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.755535 24399 raft_consensus.cc:383] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.757107 24406 leader_election.cc:290] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 election: Requested vote from peers 7679613902d342678beaffab898dfd7d (127.21.42.195:41233), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:44.755810 24399 raft_consensus.cc:738] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.758122 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf"
I20250112 08:57:44.758687 24140 raft_consensus.cc:3054] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:44.759181 24399 consensus_queue.cc:260] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.770300 24399 ts_tablet_manager.cc:1428] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 08:57:44.771292 24399 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
I20250112 08:57:44.767774 24140 raft_consensus.cc:2463] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 1.
I20250112 08:57:44.780994 23927 leader_election.cc:304] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters: 7679613902d342678beaffab898dfd7d
I20250112 08:57:44.781817 24406 raft_consensus.cc:2798] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:44.783831 24420 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:44.785447 24420 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.787376 24420 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:44.789028 24399 tablet_bootstrap.cc:654] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:44.789184 24406 raft_consensus.cc:695] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [term 1 LEADER]: Becoming Leader. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Running, Role: LEADER
I20250112 08:57:44.790170 24406 consensus_queue.cc:237] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
W20250112 08:57:44.791875 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:44.793656 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:44.794023 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:44.795142 24420 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:44.813112 24399 tablet_bootstrap.cc:492] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:44.813557 24399 ts_tablet_manager.cc:1397] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.003s
I20250112 08:57:44.816872 24399 raft_consensus.cc:357] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.817615 24399 raft_consensus.cc:383] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:44.817868 24399 raft_consensus.cc:738] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
I20250112 08:57:44.818518 24399 consensus_queue.cc:260] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:44.820653 24399 ts_tablet_manager.cc:1428] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
W20250112 08:57:44.881145 24392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:44.882017 24392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:44.882754 24392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 08:57:44.940696 24392 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 08:57:44.941040 24392 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 08:57:44.941419 24392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:57:44.941823 24392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:44.942224 24392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:57:44.942606 24392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250112 08:57:44.972678 23927 connection.cc:537] server connection from 127.0.0.1:50164 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.972918 24225 connection.cc:537] client connection to 127.21.42.253:36201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.972813 24349 connection.cc:537] client connection to 127.21.42.253:36201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.973650 24076 connection.cc:537] client connection to 127.21.42.253:36201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.974519 24222 connection.cc:537] server connection from 127.0.0.1:33160 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.974861 24349 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.42.253:36201)
W20250112 08:57:44.977206 24431 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 08:57:44.974651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.42.253:36201 (local address 127.21.42.195:38299)
0112 08:57:44.976079 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0112 08:57:44.976265 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":749}
W20250112 08:57:44.978436 23620 connection.cc:537] server connection from 127.0.0.1:41726 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.980123 24073 connection.cc:537] server connection from 127.0.0.1:44620 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:44.975783 24349 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.21.42.253:36201)
W20250112 08:57:44.984747 24349 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.21.42.253:36201)
W20250112 08:57:44.986311 24349 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.21.42.253:36201)
I20250112 08:57:44.987313 24392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39997
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.252:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.42.252
--webserver_port=38649
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:44.989044 24392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:44.991101 24392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:45.009991 24458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:45.040068 24461 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5a6dc008c89249efbae18aa4e34b9ea8)
I20250112 08:57:45.040881 24461 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:45.043776 24461 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:45.047878 23620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250112 08:57:45.052402 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:45.053135 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:45.053581 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:45.054311 24461 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 08:57:45.211423 24420 raft_consensus.cc:491] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:45.211824 24420 raft_consensus.cc:513] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:45.213363 24420 leader_election.cc:290] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:45.214442 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf" is_pre_election: true
I20250112 08:57:45.214704 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677901d662049c6b6fa75145ca455b1" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:45.215454 23994 raft_consensus.cc:2388] T 8677901d662049c6b6fa75145ca455b1 P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7679613902d342678beaffab898dfd7d in current term 1: Already voted for candidate 8f4921885fee48319c7132725c7691bf in this term.
I20250112 08:57:45.217105 24222 leader_election.cc:304] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:45.218235 24420 raft_consensus.cc:3054] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:45.222816 24420 raft_consensus.cc:2743] T 8677901d662049c6b6fa75145ca455b1 P 7679613902d342678beaffab898dfd7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:45.249318 24420 raft_consensus.cc:491] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:45.249681 24420 raft_consensus.cc:513] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:45.251245 24420 leader_election.cc:290] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:45.252475 23994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff253b3e7a14f43b118105fda65322b" is_pre_election: true
I20250112 08:57:45.252594 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4277264769840449591e82ca21bec48" candidate_uuid: "7679613902d342678beaffab898dfd7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf" is_pre_election: true
I20250112 08:57:45.253780 24140 raft_consensus.cc:2388] T b4277264769840449591e82ca21bec48 P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7679613902d342678beaffab898dfd7d in current term 1: Already voted for candidate 6ff253b3e7a14f43b118105fda65322b in this term.
I20250112 08:57:45.254817 24222 leader_election.cc:304] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:45.255491 24420 raft_consensus.cc:3054] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:45.264869 24420 raft_consensus.cc:2743] T b4277264769840449591e82ca21bec48 P 7679613902d342678beaffab898dfd7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:57:45.417784 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:45.488377 24420 consensus_queue.cc:1035] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:45.493489 24465 consensus_queue.cc:1035] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:45.539283 24465 consensus_queue.cc:1035] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:45.639940 24482 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5a6dc008c89249efbae18aa4e34b9ea8)
I20250112 08:57:45.640313 24482 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:45.641831 24482 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:45.655004 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:45.655936 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:45.656482 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:45.657474 24482 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 08:57:45.669520 24480 consensus_queue.cc:1035] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 08:57:45.695626 24480 consensus_queue.cc:1035] T b4277264769840449591e82ca21bec48 P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 08:57:45.010334 24455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:45.010429 24456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:46.313361 24457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1305 milliseconds
I20250112 08:57:46.313500 24392 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 08:57:46.317726 24392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:46.320362 24392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:46.321735 24392 hybrid_clock.cc:648] HybridClock initialized: now 1736672266321684 us; error 66 us; skew 500 ppm
I20250112 08:57:46.322535 24392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:46.333313 24392 webserver.cc:458] Webserver started at http://127.21.42.252:38649/ using document root <none> and password file <none>
I20250112 08:57:46.334340 24392 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:46.334580 24392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:46.337564 24500 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5a6dc008c89249efbae18aa4e34b9ea8)
I20250112 08:57:46.338147 24500 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.346096 24500 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:46.350106 24392 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.008s sys 0.000s
W20250112 08:57:46.352638 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:46.353310 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:46.353618 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:46.354175 24500 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250112 08:57:46.355620 24507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:46.356727 24392 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250112 08:57:46.357064 24392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
format_stamp: "Formatted at 2025-01-12 08:57:34 on dist-test-slave-ghv6"
I20250112 08:57:46.359498 24392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:46.421247 24392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:46.422770 24392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:46.423238 24392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:46.493254 24392 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.252:36643
I20250112 08:57:46.493317 24558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.252:36643 every 8 connection(s)
I20250112 08:57:46.496577 24392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-2/data/info.pb
I20250112 08:57:46.505461 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 24392
I20250112 08:57:46.512766 24559 sys_catalog.cc:263] Verifying existing consensus state
I20250112 08:57:46.523133 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Bootstrap starting.
I20250112 08:57:46.539008 24559 log.cc:826] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:46.553807 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250112 08:57:46.554565 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Bootstrap complete.
I20250112 08:57:46.574108 24559 raft_consensus.cc:357] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.576951 24559 raft_consensus.cc:738] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Initialized, Role: FOLLOWER
I20250112 08:57:46.577708 24559 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.580965 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:46.581843 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:46.583565 24559 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:57:46.627446 24579 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:46.628665 24579 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: loading cluster ID for follower catalog manager: success
I20250112 08:57:46.640621 24579 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: acquiring CA information for follower catalog manager: success
I20250112 08:57:46.646898 24579 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:46.651031 24392 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:57:46.680966 24392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:57:46.812973 24336 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:46.813474 24336 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:46.814949 24041 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:46.815480 24041 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:46.823949 24523 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" instance_seqno: 1736672258573607) as {username='slave'} at 127.21.42.193:43339; Asking this server to re-register.
I20250112 08:57:46.824414 24524 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7679613902d342678beaffab898dfd7d" instance_seqno: 1736672262685059) as {username='slave'} at 127.21.42.195:44599; Asking this server to re-register.
I20250112 08:57:46.825934 24041 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:46.826681 24336 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:46.829594 24523 ts_manager.cc:194] Registered new tserver with Master: 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:46.830611 24524 ts_manager.cc:194] Registered new tserver with Master: 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:46.858290 24185 heartbeater.cc:346] Connected to a master server at 127.21.42.252:36643
I20250112 08:57:46.858778 24185 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:46.862219 24523 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f4921885fee48319c7132725c7691bf" instance_seqno: 1736672260702202) as {username='slave'} at 127.21.42.194:55995; Asking this server to re-register.
I20250112 08:57:46.863287 24185 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:46.866416 24523 ts_manager.cc:194] Registered new tserver with Master: 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:46.911149 24591 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5a6dc008c89249efbae18aa4e34b9ea8)
I20250112 08:57:46.911547 24591 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.913123 24591 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:46.920404 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:46.925264 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" is_pre_election: true
I20250112 08:57:46.926064 24534 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53bb7660e3b04a9ca5fb697b25b17bbc in term 3.
I20250112 08:57:46.927022 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:46.927663 24591 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250112 08:57:46.928006 24591 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5a6dc008c89249efbae18aa4e34b9ea8)
I20250112 08:57:46.928359 24591 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 4 FOLLOWER]: Advancing to term 5
I20250112 08:57:46.933699 24591 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.935451 24591 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 election: Requested vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
I20250112 08:57:46.936990 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" candidate_term: 5 candidate_status { last_received { term: 4 index: 19 } } ignore_live_leader: false dest_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1"
I20250112 08:57:46.937537 24534 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 3 FOLLOWER]: Advancing to term 5
W20250112 08:57:46.945387 23620 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:46.948294 24534 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53bb7660e3b04a9ca5fb697b25b17bbc in term 5.
I20250112 08:57:46.950357 23620 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:46.951143 24591 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Leader election won for term 5
I20250112 08:57:46.952508 24591 raft_consensus.cc:695] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 LEADER]: Becoming Leader. State: Replica: 53bb7660e3b04a9ca5fb697b25b17bbc, State: Running, Role: LEADER
I20250112 08:57:46.953524 24591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 4.19, Last appended by leader: 19, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:46.958638 24595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53bb7660e3b04a9ca5fb697b25b17bbc. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:46.959410 24595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:46.961058 24597 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:57:46.967561 24597 catalog_manager.cc:670] Loaded metadata for table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]
I20250112 08:57:46.968818 24597 catalog_manager.cc:670] Loaded metadata for table default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca]
I20250112 08:57:46.969906 24597 catalog_manager.cc:670] Loaded metadata for table default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c]
I20250112 08:57:46.975045 24597 tablet_loader.cc:96] loaded metadata for tablet 8677901d662049c6b6fa75145ca455b1 (table default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca])
I20250112 08:57:46.975960 24597 tablet_loader.cc:96] loaded metadata for tablet 8c8d8e20b64b47c4a70ff492f4f11139 (table default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c])
I20250112 08:57:46.977479 24597 tablet_loader.cc:96] loaded metadata for tablet b4277264769840449591e82ca21bec48 (table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2])
I20250112 08:57:46.978422 24597 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:57:46.981397 24597 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:46.981597 24597 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:57:46.985149 24597 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:46.987962 24597 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Loaded TSK: 0
I20250112 08:57:46.988899 24597 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:46.991823 24597 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:46.995131 24597 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 08:57:47.009976 24534 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Refusing update from remote peer 53bb7660e3b04a9ca5fb697b25b17bbc: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 21. (index mismatch)
I20250112 08:57:47.012149 24595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 08:57:47.012724 24595 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [LEADER]: Peer e7c2dbf455f04f4fbcc282c86eae73e1 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20250112 08:57:47.013124 23620 consensus_peers.cc:487] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc -> Peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Couldn't send request to peer 5a6dc008c89249efbae18aa4e34b9ea8. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:47.019587 24534 pending_rounds.cc:77] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Aborting all ops after (but not including) 4
I20250112 08:57:47.019945 24534 pending_rounds.cc:99] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250112 08:57:47.020253 24534 raft_consensus.cc:2883] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 08:57:47.023423 23633 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45916:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
I20250112 08:57:47.023195 23632 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45916:
name: "default.Table_a08e8ce40f9a4a72bfd0c37e4d8b5c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:47.024413 23633 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45916:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
I20250112 08:57:47.028110 23631 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45916:
name: "default.Table_973a1a0f49a3470788ec010b2b1ceaa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:47.029423 23630 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:45916
I20250112 08:57:47.031042 23629 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45916:
name: "default.Table_49e783a6610044bc9fc98032934075db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250112 08:57:47.037304 23854 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_d4842570453f4558bc5d55fc12813c0e
*** Check failure stack trace: ***
I20250112 08:57:47.039655 23626 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45916:
table { table_name: "default.Table_64def52107a64d539a649b582e39ae08" } new_table_name: "default.Table_826bc0c69bef4c9981e5942d7127fb86" modify_external_catalogs: true
I20250112 08:57:47.041764 23627 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45916:
table { table_name: "default.Table_7c040710b44a4de7bb1a466c66def679" } new_table_name: "default.Table_34ded4956e99401f95167457615dd003" modify_external_catalogs: true
I20250112 08:57:47.043478 23628 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:45916
I20250112 08:57:47.048688 23630 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45916:
name: "default.Table_8b0ef96b104f4089b1023146b02467bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fbb36285950 google::LogMessage::Flush() at ??:0
I20250112 08:57:47.067394 24565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53bb7660e3b04a9ca5fb697b25b17bbc. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:47.068172 24565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:47.071308 24598 mvcc.cc:204] Tried to move back new op lower bound from 7113409605657616384 to 7113409605451378688. Current Snapshot: MvccSnapshot[applied={T|T < 7113409605657616384}]
I20250112 08:57:47.071136 24601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:47.072254 24601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:47.073093 24601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:47.075871 24601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:47.075044 24602 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:47.077522 24602 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:47.086937 24595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:47.087713 24595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: LEADER
@ 0x7fbb362856bf google::LogMessage::~LogMessage() at ??:0
I20250112 08:57:47.116500 24599 mvcc.cc:204] Tried to move back new op lower bound from 7113409605657616384 to 7113409585574133760. Current Snapshot: MvccSnapshot[applied={T|T < 7113409595476897792}]
@ 0x7fbb4118082a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fbb4118007c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fbb41183d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fbb41183cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fbb41183c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fbb41183c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fbb41182f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
@ 0x7fbb384ca8d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250112 08:57:47.245195 23628 master_service.cc:946] ReplaceTablet: received request to replace tablet b4277264769840449591e82ca21bec48 from {username='slave'} at 127.0.0.1:45916
@ 0x7fbb39b496db start_thread at ??:0
I20250112 08:57:47.269109 23628 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Sending DeleteTablet for 3 replicas of tablet b4277264769840449591e82ca21bec48
@ 0x7fbb34f2a71f clone at ??:0
W20250112 08:57:47.485337 24346 connection.cc:537] client connection to 127.21.42.254:41739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:47.485723 24222 connection.cc:537] client connection to 127.21.42.254:41739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:47.486685 24073 connection.cc:537] client connection to 127.21.42.254:41739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:47.487337 24346 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.21.42.254:41739)
W20250112 08:57:47.488225 23927 connection.cc:537] client connection to 127.21.42.254:41739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:47.489801 24346 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.21.42.254:41739)
W20250112 08:57:47.491014 24346 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.42.254:41739)
W20250112 08:57:47.492777 24346 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.21.42.254:41739)
I20250112 08:57:47.767036 24622 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:47.767473 24622 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:47.770660 24622 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:47.781587 24511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:47.787981 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:47.788489 24508 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739): Network error: Client connection negotiation failed: client connection to 127.21.42.254:41739: connect: Connection refused (error 111)
I20250112 08:57:47.788883 24508 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:47.789602 24622 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250112 08:57:48.143072 24631 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:48.143570 24631 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:48.145826 24631 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:57:48.146976 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:48.147418 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:48.147739 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:48.148442 24631 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:48.533273 24637 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:48.533641 24637 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:48.535085 24637 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:48.541838 24508 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739): Network error: Client connection negotiation failed: client connection to 127.21.42.254:41739: connect: Connection refused (error 111)
W20250112 08:57:48.542332 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:48.542620 24511 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:48.543185 24637 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250112 08:57:49.146942 24649 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:49.147323 24649 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:49.148828 24649 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:49.152983 24508 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739): Network error: Client connection negotiation failed: client connection to 127.21.42.254:41739: connect: Connection refused (error 111)
W20250112 08:57:49.153578 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:49.153894 24511 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:49.154410 24649 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250112 08:57:49.494496 21675 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
/tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:41739
--webserver_interface=127.21.42.254
--webserver_port=42815
--builtin_ntp_servers=127.21.42.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--hive_metastore_uris=thrift://127.0.0.1:39997
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 08:57:49.842296 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 08:57:49.842921 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 08:57:49.843400 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 08:57:49.844647 24664 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:49.845037 24664 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:49.846985 24664 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:49.854139 24508 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739): Network error: Client connection negotiation failed: client connection to 127.21.42.254:41739: connect: Connection refused (error 111)
W20250112 08:57:49.870700 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:49.871235 24511 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:49.872545 24664 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250112 08:57:49.889755 24658 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 08:57:49.890014 24658 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 08:57:49.890368 24658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 08:57:49.890610 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 08:57:49.890846 24658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 08:57:49.891120 24658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 08:57:49.929944 24658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.42.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39997
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.42.254:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.42.254
--webserver_port=42815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 08:43:20 UTC on a633016ff5c9
build id 4140
TSAN enabled
I20250112 08:57:49.931344 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 08:57:49.932978 24658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 08:57:49.950937 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:50.724610 24689 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:50.725100 24689 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:50.744745 24689 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:50.759387 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:50.761749 24508 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739): Network error: Client connection negotiation failed: client connection to 127.21.42.254:41739: connect: Connection refused (error 111)
I20250112 08:57:50.762205 24508 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 53bb7660e3b04a9ca5fb697b25b17bbc, 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:50.762959 24689 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250112 08:57:49.951030 24674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 08:57:49.952061 24676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 08:57:49.952795 24658 server_base.cc:1034] running on GCE node
I20250112 08:57:51.197741 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 08:57:51.200485 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 08:57:51.201848 24658 hybrid_clock.cc:648] HybridClock initialized: now 1736672271201815 us; error 30 us; skew 500 ppm
I20250112 08:57:51.202638 24658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 08:57:51.209004 24658 webserver.cc:458] Webserver started at http://127.21.42.254:42815/ using document root <none> and password file <none>
I20250112 08:57:51.210084 24658 fs_manager.cc:362] Metadata directory not provided
I20250112 08:57:51.210320 24658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 08:57:51.218072 24658 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20250112 08:57:51.222782 24702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 08:57:51.224126 24658 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250112 08:57:51.224432 24658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data,/tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
format_stamp: "Formatted at 2025-01-12 08:57:30 on dist-test-slave-ghv6"
I20250112 08:57:51.226699 24658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 08:57:51.284617 24658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 08:57:51.286383 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 08:57:51.286828 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 08:57:51.358644 24658 rpc_server.cc:307] RPC server started. Bound to: 127.21.42.254:41739
I20250112 08:57:51.358815 24754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.42.254:41739 every 8 connection(s)
I20250112 08:57:51.361581 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTU_TaI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736672197602273-21675-0/minicluster-data/master-0/data/info.pb
I20250112 08:57:51.370157 21675 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTU_TaI/build/tsan/bin/kudu as pid 24658
I20250112 08:57:51.372658 24755 sys_catalog.cc:263] Verifying existing consensus state
I20250112 08:57:51.383265 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Bootstrap starting.
I20250112 08:57:51.415834 24755 log.cc:826] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Log is configured to *not* fsync() on all Append() calls
I20250112 08:57:51.416640 24045 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:51.470980 24338 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:51.472226 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=13 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 08:57:51.473124 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: Bootstrap complete.
I20250112 08:57:51.493165 24755 raft_consensus.cc:357] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:51.496119 24755 raft_consensus.cc:738] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bb7660e3b04a9ca5fb697b25b17bbc, State: Initialized, Role: FOLLOWER
I20250112 08:57:51.496852 24755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 5.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:51.499266 24765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:51.500088 24765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:51.500263 24755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: configured and running, proceeding with master startup.
I20250112 08:57:51.518757 24187 heartbeater.cc:346] Connected to a master server at 127.21.42.254:41739
I20250112 08:57:51.553597 24780 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:51.553925 24780 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: loading cluster ID for follower catalog manager: success
I20250112 08:57:51.562578 24780 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: acquiring CA information for follower catalog manager: success
I20250112 08:57:51.570802 24780 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 08:57:51.554045 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:57:51.582254 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:57:51.604578 24783 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:51.605023 24783 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:51.606750 24783 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
W20250112 08:57:51.615880 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:51.620052 24730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" is_pre_election: true
I20250112 08:57:51.621116 24730 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 5.
I20250112 08:57:51.622387 24508 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:51.623169 24783 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250112 08:57:51.623430 24783 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 53bb7660e3b04a9ca5fb697b25b17bbc)
I20250112 08:57:51.623677 24783 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 5 FOLLOWER]: Advancing to term 6
I20250112 08:57:51.628279 24783 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:51.629849 24783 leader_election.cc:290] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 election: Requested vote from peers 53bb7660e3b04a9ca5fb697b25b17bbc (127.21.42.254:41739), 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201)
I20250112 08:57:51.630587 24730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: false dest_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc"
I20250112 08:57:51.631054 24730 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 5 FOLLOWER]: Advancing to term 6
W20250112 08:57:51.634527 24511 leader_election.cc:336] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:51.638586 24730 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c2dbf455f04f4fbcc282c86eae73e1 in term 6.
I20250112 08:57:51.639557 24508 leader_election.cc:304] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc, e7c2dbf455f04f4fbcc282c86eae73e1; no voters: 5a6dc008c89249efbae18aa4e34b9ea8
I20250112 08:57:51.640713 24783 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 6 FOLLOWER]: Leader election won for term 6
I20250112 08:57:51.642083 24783 raft_consensus.cc:695] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [term 6 LEADER]: Becoming Leader. State: Replica: e7c2dbf455f04f4fbcc282c86eae73e1, State: Running, Role: LEADER
I20250112 08:57:51.643090 24783 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 5.22, Last appended by leader: 22, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:51.647624 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 6 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:51.648434 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:51.650951 24789 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 08:57:51.656841 24789 catalog_manager.cc:670] Loaded metadata for table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]
I20250112 08:57:51.658082 24789 catalog_manager.cc:670] Loaded metadata for table default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca]
I20250112 08:57:51.659359 24789 catalog_manager.cc:670] Loaded metadata for table default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c]
I20250112 08:57:51.664484 24789 tablet_loader.cc:96] loaded metadata for tablet 49c9ffe06acd43438d01b9341d4df7dc (table default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2])
I20250112 08:57:51.665470 24789 tablet_loader.cc:96] loaded metadata for tablet 8677901d662049c6b6fa75145ca455b1 (table default.table_d16629577a884abab7f9b83a12d4bc97 [id=ad737da0fbd64899bf4d28c3199895ca])
I20250112 08:57:51.666548 24789 tablet_loader.cc:96] loaded metadata for tablet 8c8d8e20b64b47c4a70ff492f4f11139 (table default.table_7c040710b44a4de7bb1a466c66def679 [id=d8ab19d75e784220b1c16aca0d90678c])
I20250112 08:57:51.668212 24789 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 08:57:51.672541 24789 catalog_manager.cc:1260] Loaded cluster ID: b1ddd673261d4304b0e4f908c5224ef7
I20250112 08:57:51.672791 24789 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 08:57:51.679244 24789 catalog_manager.cc:1505] Loading token signing keys...
I20250112 08:57:51.683720 24789 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1: Loaded TSK: 0
I20250112 08:57:51.685108 24789 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 08:57:51.689328 24392 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:57:51.689736 24392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:57:51.690997 24789 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 08:57:51.694689 24789 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 08:57:51.714864 24730 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Refusing update from remote peer e7c2dbf455f04f4fbcc282c86eae73e1: Log matching property violated. Preceding OpId in replica: term: 5 index: 22. Preceding OpId from leader: term: 6 index: 24. (index mismatch)
I20250112 08:57:51.716423 24783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20250112 08:57:51.717155 24511 consensus_peers.cc:487] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 -> Peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Couldn't send request to peer 5a6dc008c89249efbae18aa4e34b9ea8. Status: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 08:57:51.734508 24765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c2dbf455f04f4fbcc282c86eae73e1. Latest consensus state: current_term: 6 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:51.735241 24765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:51.742708 24792 mvcc.cc:204] Tried to move back new op lower bound from 7113409624928387072 to 7113409624660107264. Current Snapshot: MvccSnapshot[applied={T|T < 7113409624928387072}]
I20250112 08:57:51.751045 24765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:51.752076 24765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [sys.catalog]: This master's current role is: FOLLOWER
I20250112 08:57:51.824429 24120 tablet_service.cc:1467] Processing CreateTablet for tablet 49c9ffe06acd43438d01b9341d4df7dc (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:51.825791 24120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c9ffe06acd43438d01b9341d4df7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:51.841439 24524 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35802
I20250112 08:57:51.842741 23973 tablet_service.cc:1467] Processing CreateTablet for tablet 49c9ffe06acd43438d01b9341d4df7dc (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:51.843668 24524 master_service.cc:946] ReplaceTablet: received request to replace tablet b4277264769840449591e82ca21bec48 from {username='slave'} at 127.0.0.1:35802
I20250112 08:57:51.844954 23973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c9ffe06acd43438d01b9341d4df7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:51.845327 24523 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "default.Table_64def52107a64d539a649b582e39ae08" } new_table_name: "default.Table_826bc0c69bef4c9981e5942d7127fb86" modify_external_catalogs: true
I20250112 08:57:51.847546 24522 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "default.Table_7c040710b44a4de7bb1a466c66def679" } new_table_name: "default.Table_34ded4956e99401f95167457615dd003" modify_external_catalogs: true
I20250112 08:57:51.852023 24520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "default.Table_49e783a6610044bc9fc98032934075db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:51.851881 24521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "default.Table_8b0ef96b104f4089b1023146b02467bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250112 08:57:51.859411 24578 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_d4842570453f4558bc5d55fc12813c0e
I20250112 08:57:51.860220 24519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "default.Table_973a1a0f49a3470788ec010b2b1ceaa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:51.860733 24518 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
I20250112 08:57:51.865823 24518 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35802:
table { table_name: "default.Table_d16629577a884abab7f9b83a12d4bc97" } modify_external_catalogs: true
*** Check failure stack trace: ***
I20250112 08:57:51.862694 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } } }
I20250112 08:57:51.869977 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c2dbf455f04f4fbcc282c86eae73e1 [sys.catalog]: This master's current role is: LEADER
I20250112 08:57:51.863209 24517 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35802:
name: "default.Table_a08e8ce40f9a4a72bfd0c37e4d8b5c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 08:57:51.874262 24271 tablet_service.cc:1467] Processing CreateTablet for tablet 49c9ffe06acd43438d01b9341d4df7dc (DEFAULT_TABLE table=default.table_64def52107a64d539a649b582e39ae08 [id=0302d2aee6434879a8f4e7fe405748c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 08:57:51.875581 24271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c9ffe06acd43438d01b9341d4df7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 08:57:51.883607 24801 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf: Bootstrap starting.
I20250112 08:57:51.890754 24801 tablet_bootstrap.cc:654] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:51.901634 24802 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b: Bootstrap starting.
I20250112 08:57:51.904050 24801 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf: No bootstrap required, opened a new log
I20250112 08:57:51.904582 24801 ts_tablet_manager.cc:1397] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20250112 08:57:51.907294 24801 raft_consensus.cc:357] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.908368 24801 raft_consensus.cc:383] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:51.909186 24801 raft_consensus.cc:738] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4921885fee48319c7132725c7691bf, State: Initialized, Role: FOLLOWER
I20250112 08:57:51.909528 24802 tablet_bootstrap.cc:654] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:51.910576 24801 consensus_queue.cc:260] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.911756 24804 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d: Bootstrap starting.
I20250112 08:57:51.915302 24801 ts_tablet_manager.cc:1428] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
@ 0x7f210b240950 google::LogMessage::Flush() at ??:0
I20250112 08:57:51.921015 24804 tablet_bootstrap.cc:654] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d: Neither blocks nor log segments found. Creating new log.
I20250112 08:57:51.926189 24719 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" instance_seqno: 1736672258573607) as {username='slave'} at 127.21.42.193:60789; Asking this server to re-register.
I20250112 08:57:51.919332 24185 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:51.928798 24045 heartbeater.cc:463] Registering TS with master...
@ 0x7f210b2406bf google::LogMessage::~LogMessage() at ??:0
I20250112 08:57:51.932224 24720 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f4921885fee48319c7132725c7691bf" instance_seqno: 1736672260702202) as {username='slave'} at 127.21.42.194:34169; Asking this server to re-register.
W20250112 08:57:51.933575 24524 catalog_manager.cc:5359] Ignoring report from unknown tablet b4277264769840449591e82ca21bec48
I20250112 08:57:51.933612 24187 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:51.934172 24719 ts_manager.cc:194] Registered new tserver with Master: 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101)
I20250112 08:57:51.936403 24524 catalog_manager.cc:5526] T 8677901d662049c6b6fa75145ca455b1 P 8f4921885fee48319c7132725c7691bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4921885fee48319c7132725c7691bf (127.21.42.194). New cstate: current_term: 1 leader_uuid: "8f4921885fee48319c7132725c7691bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } health_report { overall_health: HEALTHY } } }
I20250112 08:57:51.938616 24719 ts_manager.cc:194] Registered new tserver with Master: 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
I20250112 08:57:51.939945 24804 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d: No bootstrap required, opened a new log
I20250112 08:57:51.940392 24804 ts_tablet_manager.cc:1397] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.004s
I20250112 08:57:51.943224 24804 raft_consensus.cc:357] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.944124 24804 raft_consensus.cc:383] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:51.944407 24804 raft_consensus.cc:738] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7679613902d342678beaffab898dfd7d, State: Initialized, Role: FOLLOWER
I20250112 08:57:51.945076 24804 consensus_queue.cc:260] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.947607 24804 ts_tablet_manager.cc:1428] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250112 08:57:51.954530 24719 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7679613902d342678beaffab898dfd7d" instance_seqno: 1736672262685059) as {username='slave'} at 127.21.42.195:49263; Asking this server to re-register.
I20250112 08:57:51.956609 24338 heartbeater.cc:463] Registering TS with master...
I20250112 08:57:51.957937 24802 tablet_bootstrap.cc:492] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b: No bootstrap required, opened a new log
I20250112 08:57:51.958397 24802 ts_tablet_manager.cc:1397] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.005s
I20250112 08:57:51.960546 24719 ts_manager.cc:194] Registered new tserver with Master: 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:51.961192 24802 raft_consensus.cc:357] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.961941 24802 raft_consensus.cc:383] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 08:57:51.962229 24802 raft_consensus.cc:738] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Initialized, Role: FOLLOWER
I20250112 08:57:51.963116 24802 consensus_queue.cc:260] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:51.965220 24041 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
I20250112 08:57:51.965898 24802 ts_tablet_manager.cc:1428] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
@ 0x7f211613b82a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
W20250112 08:57:51.974956 24515 catalog_manager.cc:5359] Ignoring report from unknown tablet b4277264769840449591e82ca21bec48
W20250112 08:57:51.975497 24515 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
I20250112 08:57:51.981319 24336 heartbeater.cc:502] Master 127.21.42.252:36643 was elected leader, sending a full tablet report...
W20250112 08:57:51.985961 24524 catalog_manager.cc:5359] Ignoring report from unknown tablet b4277264769840449591e82ca21bec48
W20250112 08:57:51.986459 24524 catalog_manager.cc:5359] Ignoring report from unknown tablet 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:51.986690 24524 catalog_manager.cc:5359] Ignoring report from unknown tablet 77a9b2f822354182b0e36c55cce850ce
@ 0x7f211613b07c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f211613ed32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f211613ecea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7f211613ec7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f211613ec42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250112 08:57:52.069833 24516 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35802
@ 0x7f211613df3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250112 08:57:52.091203 24516 master_service.cc:946] ReplaceTablet: received request to replace tablet 77a9b2f822354182b0e36c55cce850ce from {username='slave'} at 127.0.0.1:35802
@ 0x7f210d4858d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
@ 0x7f210eb046db start_thread at ??:0
@ 0x7f2109ee571f clone at ??:0
W20250112 08:57:52.315631 24349 connection.cc:537] client connection to 127.21.42.252:36643 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 08:57:52.317476 24349 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.21.42.252:36643)
W20250112 08:57:52.318694 24349 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.42.252:36643)
W20250112 08:57:52.319353 24349 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.21.42.252:36643)
W20250112 08:57:52.321305 24349 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.21.42.252:36643)
I20250112 08:57:52.355984 24707 raft_consensus.cc:491] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:52.356491 24707 raft_consensus.cc:513] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:52.359886 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c9ffe06acd43438d01b9341d4df7dc" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf" is_pre_election: true
I20250112 08:57:52.359939 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c9ffe06acd43438d01b9341d4df7dc" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d" is_pre_election: true
I20250112 08:57:52.360668 24140 raft_consensus.cc:2463] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 0.
I20250112 08:57:52.360694 24291 raft_consensus.cc:2463] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 0.
I20250112 08:57:52.358598 24707 leader_election.cc:290] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375), 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:52.361987 23927 leader_election.cc:304] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters:
I20250112 08:57:52.363595 24707 raft_consensus.cc:2798] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 08:57:52.364022 24707 raft_consensus.cc:491] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 08:57:52.364360 24707 raft_consensus.cc:3054] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:52.371204 24707 raft_consensus.cc:513] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:52.374224 24707 leader_election.cc:290] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 election: Requested vote from peers 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375), 7679613902d342678beaffab898dfd7d (127.21.42.195:41233)
I20250112 08:57:52.375331 24140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c9ffe06acd43438d01b9341d4df7dc" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4921885fee48319c7132725c7691bf"
I20250112 08:57:52.375571 24291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c9ffe06acd43438d01b9341d4df7dc" candidate_uuid: "6ff253b3e7a14f43b118105fda65322b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7679613902d342678beaffab898dfd7d"
I20250112 08:57:52.375929 24140 raft_consensus.cc:3054] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:52.376173 24291 raft_consensus.cc:3054] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Advancing to term 1
I20250112 08:57:52.382459 24140 raft_consensus.cc:2463] T 49c9ffe06acd43438d01b9341d4df7dc P 8f4921885fee48319c7132725c7691bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 1.
I20250112 08:57:52.383742 23927 leader_election.cc:304] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf; no voters:
I20250112 08:57:52.384255 24291 raft_consensus.cc:2463] T 49c9ffe06acd43438d01b9341d4df7dc P 7679613902d342678beaffab898dfd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff253b3e7a14f43b118105fda65322b in term 1.
I20250112 08:57:52.386010 24707 raft_consensus.cc:2798] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 08:57:52.393200 24707 raft_consensus.cc:695] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [term 1 LEADER]: Becoming Leader. State: Replica: 6ff253b3e7a14f43b118105fda65322b, State: Running, Role: LEADER
I20250112 08:57:52.395423 24707 consensus_queue.cc:237] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } } peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } }
I20250112 08:57:52.466094 24826 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:52.466531 24826 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:52.469156 24826 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:52.475574 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:52.480865 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:52.481555 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:52.481966 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:52.484014 24826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:57:53.028679 24831 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:53.029070 24831 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:53.030627 24831 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:53.037624 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:53.038357 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:53.038667 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:53.039212 24831 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:57:53.329118 24835 consensus_queue.cc:1035] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 08:57:53.355391 24838 consensus_queue.cc:1035] T 49c9ffe06acd43438d01b9341d4df7dc P 6ff253b3e7a14f43b118105fda65322b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 08:57:53.811775 21675 master-stress-test.cc:577] Killing test thread 1/10
I20250112 08:57:53.820760 24849 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:53.821152 24849 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:53.822633 24849 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:53.827373 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:53.827957 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:53.828343 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:53.828910 24849 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:57:53.854060 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5d40 after lost signal to thread 24197
W20250112 08:57:53.854895 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000d45e0 after lost signal to thread 24335
I20250112 08:57:54.611186 24863 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:54.611702 24863 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:54.613242 24863 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:57:54.615242 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:54.615996 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:54.616289 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:54.616852 24863 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:54.856540 24868 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:54.856926 24868 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:54.858457 24868 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:54.866541 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:57:54.867012 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
I20250112 08:57:54.867297 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:54.867811 24868 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:57:55.418345 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 6: this message will repeat every 5th retry.
I20250112 08:57:56.588181 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:57:56.588562 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:57:57.189536 24903 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:57.189944 24903 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:57.191509 24903 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:57.196367 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:57.197080 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:57.197403 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:57.197963 24903 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:57:59.207381 24941 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:57:59.207767 24941 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:57:59.209309 24941 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:57:59.210909 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:57:59.211477 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:57:59.211767 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:57:59.212352 24941 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:57:59.696794 24949 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:57:59.697340 24949 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:57:59.699028 24949 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:57:59.701188 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250112 08:57:59.703576 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:57:59.704294 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:57:59.704576 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:57:59.705137 24949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:01.595131 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:01.595650 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:04.771720 25039 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:04.772225 25039 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:04.773820 25039 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:04.776116 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:58:04.779067 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:04.779685 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:04.780004 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:04.780683 25039 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:58:05.179476 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 11: this message will repeat every 5th retry.
I20250112 08:58:05.391165 25050 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:58:05.391709 25050 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:58:05.394001 25050 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:58:05.395514 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:58:05.395984 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:58:05.396240 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:58:05.396773 25050 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:58:06.600852 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:06.601200 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:08.206894 25102 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:08.207289 25102 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:08.208871 25102 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:08.213321 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:08.214046 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:08.214311 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:08.214761 25102 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:09.694257 25128 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:09.694638 25128 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:09.696072 25128 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:09.700969 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:09.701634 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:09.701908 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:09.702411 25128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:11.608001 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:11.608572 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:12.282480 25176 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:12.282990 25176 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:12.284654 25176 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:12.286676 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250112 08:58:12.289033 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:12.289793 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:12.290047 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:12.290570 25176 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:13.337527 25189 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:58:13.338048 25189 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:58:13.339679 25189 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:58:13.341349 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:58:13.341858 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:58:13.342132 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:58:13.342672 25189 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:58:14.780819 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 16: this message will repeat every 5th retry.
I20250112 08:58:16.613600 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:16.613924 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:21.619374 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:21.619750 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:58:22.408444 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb020 after lost signal to thread 24053
W20250112 08:58:22.409526 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000457a0 after lost signal to thread 24184
W20250112 08:58:25.035111 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 21: this message will repeat every 5th retry.
I20250112 08:58:26.624588 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:26.624922 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:26.754093 25412 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:26.754606 25412 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:26.756470 25412 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:26.758198 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:58:26.761060 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:26.761761 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:26.762030 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:26.762509 25412 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:28.785281 25448 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:58:28.785919 25448 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:58:28.788045 25448 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:58:28.790002 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:58:28.790613 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:58:28.790992 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:58:28.791549 25448 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:58:31.630525 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:31.630928 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:58:34.392750 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7100 after lost signal to thread 23900
W20250112 08:58:34.393626 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000de260 after lost signal to thread 24040
W20250112 08:58:35.577716 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 26: this message will repeat every 5th retry.
W20250112 08:58:35.905004 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd6e0 after lost signal to thread 24660
W20250112 08:58:35.905838 24751 debug-util.cc:398] Leaking SignalData structure 0x7b0800073700 after lost signal to thread 24754
I20250112 08:58:36.662711 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:36.663111 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:41.669001 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:41.669443 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:58:43.380129 24337 heartbeater.cc:643] Failed to heartbeat to 127.21.42.253:36201 (151 consecutive failures): Network error: Failed to ping master at 127.21.42.253:36201: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:43.589160 24041 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (103 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:58:43.734647 24185 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (103 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:44.055282 25702 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:44.055995 25702 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:44.058553 25702 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:44.063359 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:58:44.073567 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:44.074385 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:44.074730 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:44.075390 25702 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:58:46.116807 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 31: this message will repeat every 5th retry.
I20250112 08:58:46.674178 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:46.674520 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:48.158010 25774 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:58:48.158542 25774 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:58:48.160504 25774 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:58:48.162067 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:58:48.162595 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:58:48.162918 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:58:48.163563 25774 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:58:51.680311 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:51.680727 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:58:53.117157 25862 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:53.117606 25862 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:53.119131 25862 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:53.122438 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:58:53.125542 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:53.126152 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:53.126487 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:53.127171 25862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:58:55.667587 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 36: this message will repeat every 5th retry.
I20250112 08:58:56.620738 25924 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:58:56.621220 25924 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:58:56.622650 25924 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:58:56.626731 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:58:56.627458 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:58:56.627714 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:58:56.628208 25924 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:58:56.684773 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:58:56.685204 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:01.689787 26009 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:59:01.690402 26009 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:59:01.691025 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:01.691447 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:01.692287 26009 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:59:01.693790 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:59:01.694272 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:59:01.694563 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:59:01.695053 26009 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 08:59:05.384871 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 41: this message will repeat every 5th retry.
I20250112 08:59:06.696291 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:06.696653 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:11.702664 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:11.703150 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:13.178280 26200 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:59:13.178762 26200 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:59:13.180703 26200 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:59:13.182883 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 08:59:13.185422 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:59:13.186158 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:59:13.186443 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:59:13.187005 26200 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 08:59:15.409675 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 46: this message will repeat every 5th retry.
I20250112 08:59:16.708259 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:16.708616 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:17.296496 26270 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:59:17.296995 26270 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:59:17.299000 26270 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:59:17.300545 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:59:17.300921 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:59:17.301175 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:59:17.301723 26270 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:59:21.714509 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:21.714946 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:59:25.605309 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 51: this message will repeat every 5th retry.
I20250112 08:59:26.720077 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:26.720409 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:27.036067 26437 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:59:27.036585 26437 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:59:27.038154 26437 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:59:27.039822 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:59:27.040318 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:59:27.040580 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:59:27.041150 26437 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:59:28.047348 26451 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:59:28.047880 26451 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:59:28.049644 26451 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:59:28.052696 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:59:28.055621 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:59:28.056087 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:59:28.056330 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:59:28.056833 26451 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:59:31.726009 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:31.726588 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:59:33.068706 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc640 after lost signal to thread 24660
W20250112 08:59:33.069406 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000428a0 after lost signal to thread 24754
W20250112 08:59:35.312028 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 56: this message will repeat every 5th retry.
I20250112 08:59:36.763458 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:36.763845 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:59:39.056280 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000dfe60 after lost signal to thread 24197
W20250112 08:59:39.057333 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba7a0 after lost signal to thread 24335
I20250112 08:59:40.114795 26654 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:59:40.115355 26654 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:59:40.117538 26654 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:59:40.119235 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:59:40.119608 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:59:40.119876 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:59:40.120368 26654 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:59:40.768137 26667 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:59:40.768739 26667 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:59:40.770426 26667 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:59:40.772768 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:59:40.776023 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:59:40.776648 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:59:40.776885 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:59:40.777357 26667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:59:41.769421 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:41.769798 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:59:43.427407 24336 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (276 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:59:43.999665 24041 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (223 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:59:44.216579 24185 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (223 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 08:59:45.374099 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 61: this message will repeat every 5th retry.
I20250112 08:59:46.774713 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:46.775045 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:49.623032 26815 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 08:59:49.623515 26815 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 08:59:49.625146 26815 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 08:59:49.626767 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 08:59:49.627194 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 08:59:49.627496 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 08:59:49.628037 26815 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 08:59:51.613418 24709 maintenance_manager.cc:418] P 53bb7660e3b04a9ca5fb697b25b17bbc: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033365
I20250112 08:59:51.682556 24708 maintenance_manager.cc:642] P 53bb7660e3b04a9ca5fb697b25b17bbc: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.068s user 0.027s sys 0.023s Metrics: {"bytes_written":9154,"cfile_init":1,"drs_written":1,"lbm_read_time_us":201,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2203,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":13,"thread_start_us":521,"threads_started":1,"wal-append.queue_time_us":836}
I20250112 08:59:51.684696 24709 maintenance_manager.cc:418] P 53bb7660e3b04a9ca5fb697b25b17bbc: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 845 bytes on disk
I20250112 08:59:51.686305 24708 maintenance_manager.cc:642] P 53bb7660e3b04a9ca5fb697b25b17bbc: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":231,"lbm_reads_lt_1ms":4}
I20250112 08:59:51.780639 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:51.781065 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 08:59:55.866472 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 66: this message will repeat every 5th retry.
I20250112 08:59:56.786280 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 08:59:56.786655 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 08:59:56.944842 26939 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:59:56.945300 26939 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:59:56.947105 26939 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:59:56.948980 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 08:59:56.951970 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:59:56.952540 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:59:56.952771 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:59:56.953264 26939 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 08:59:57.551163 26952 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 08:59:57.551504 26952 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 08:59:57.552911 26952 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 08:59:57.557976 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 08:59:57.558507 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 08:59:57.558766 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 08:59:57.559434 26952 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:00:00.744122 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000dba20 after lost signal to thread 24053
W20250112 09:00:00.744973 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5500 after lost signal to thread 24184
I20250112 09:00:01.792923 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:01.793577 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:04.980140 27097 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:04.980688 27097 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:04.982383 27097 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:04.984843 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:00:04.987860 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:04.988528 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:04.988799 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:04.989323 27097 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:00:06.497774 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 71: this message will repeat every 5th retry.
I20250112 09:00:06.799213 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:06.799654 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:07.068948 27133 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:00:07.069507 27133 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:00:07.071836 27133 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:00:07.073566 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:00:07.074008 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:00:07.074312 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:00:07.074836 27133 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:00:11.806128 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:11.806687 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:12.197052 27250 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:12.197608 27250 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:12.199316 27250 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:12.201059 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:00:12.204237 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:12.204813 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:12.205063 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:12.205578 27250 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:00:12.534150 27259 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:00:12.534688 27259 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:00:12.536429 27259 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:00:12.537920 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:00:12.538489 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:00:12.538739 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:00:12.539247 27259 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:00:14.532871 27301 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:14.533304 27301 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:14.534986 27301 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:14.539499 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:14.540083 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:14.540346 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:14.540797 27301 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:00:15.245400 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000df8a0 after lost signal to thread 24197
W20250112 09:00:15.246196 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000dfe00 after lost signal to thread 24335
W20250112 09:00:16.498328 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 76: this message will repeat every 5th retry.
I20250112 09:00:16.813931 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:16.814389 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:21.821379 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:21.821955 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:26.827484 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:26.827870 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:00:26.833420 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 81: this message will repeat every 5th retry.
I20250112 09:00:27.104919 27554 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:27.105463 27554 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:27.107110 27554 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:27.109068 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:00:27.112354 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:27.112972 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:27.113231 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:27.113756 27554 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:00:28.333115 27582 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:00:28.333665 27582 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:00:28.336181 27582 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:00:28.337720 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:00:28.338099 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:00:28.338351 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:00:28.338858 27582 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 09:00:29.851410 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000d28e0 after lost signal to thread 23900
W20250112 09:00:29.852365 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000d26c0 after lost signal to thread 24040
W20250112 09:00:30.468982 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000cce80 after lost signal to thread 24197
W20250112 09:00:30.469851 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6a60 after lost signal to thread 24335
I20250112 09:00:30.553035 27629 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:30.553696 27629 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:30.556016 27629 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:30.561404 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:30.562038 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:30.562355 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:30.563033 27629 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:00:31.154202 24037 debug-util.cc:398] Leaking SignalData structure 0x7b080005e100 after lost signal to thread 23900
W20250112 09:00:31.155054 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000cac80 after lost signal to thread 24040
I20250112 09:00:31.833453 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:31.833863 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:36.838829 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:36.839167 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:00:36.923797 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 86: this message will repeat every 5th retry.
I20250112 09:00:39.208575 27802 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:00:39.209074 27802 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:00:39.210824 27802 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:00:39.212394 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:00:39.212834 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:00:39.213100 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:00:39.213629 27802 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:00:41.845336 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:41.845785 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:00:43.775559 24337 heartbeater.cc:643] Failed to heartbeat to 127.21.42.253:36201 (451 consecutive failures): Network error: Failed to ping master at 127.21.42.253:36201: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:44.469530 24041 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (343 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:00:44.613245 24185 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (343 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:00:46.221024 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 91: this message will repeat every 5th retry.
I20250112 09:00:46.851326 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:46.851660 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:48.493115 27989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:48.493597 27989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:48.495546 27989 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:48.498984 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:00:48.508157 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:48.510612 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:48.511104 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:48.511876 27989 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:00:51.858052 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:51.858455 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:54.470659 28111 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:00:54.471210 28111 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:00:54.473397 28111 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:00:54.475060 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:00:54.475528 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:00:54.475842 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:00:54.476397 28111 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 09:00:56.298840 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 96: this message will repeat every 5th retry.
I20250112 09:00:56.863595 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:00:56.863992 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:00:59.988107 28223 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:00:59.988644 28223 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:00:59.990422 28223 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:00:59.992990 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:00:59.997581 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:00:59.998214 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:00:59.998485 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:00:59.999176 28223 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:01:01.869541 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:01.869964 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:01:06.219378 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 101: this message will repeat every 5th retry.
I20250112 09:01:06.875267 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:06.875638 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:08.333652 28386 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:01:08.334322 28386 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:01:08.336800 28386 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:01:08.338585 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:01:08.339098 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:01:08.339336 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:01:08.339843 28386 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:01:11.881429 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:11.881868 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:01:14.038739 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccb60 after lost signal to thread 24197
W20250112 09:01:14.039731 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1700 after lost signal to thread 24335
W20250112 09:01:16.618510 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 106: this message will repeat every 5th retry.
I20250112 09:01:16.887219 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:16.887560 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:18.134514 28584 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:01:18.135020 28584 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:01:18.137538 28584 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:01:18.139963 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:01:18.143071 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:18.143692 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:01:18.143977 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:01:18.144524 28584 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:01:21.067401 28645 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:01:21.068045 28645 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:01:21.069864 28645 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:01:21.075673 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:21.076367 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:01:21.076630 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:01:21.077183 28645 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:01:21.893244 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:21.893694 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:24.243800 28711 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:01:24.244426 28711 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:01:24.246675 28711 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:01:24.248315 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:01:24.248747 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:01:24.249104 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:01:24.249696 28711 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 09:01:24.636409 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4100 after lost signal to thread 24660
W20250112 09:01:24.637018 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000b67e0 after lost signal to thread 24754
W20250112 09:01:26.376446 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 111: this message will repeat every 5th retry.
W20250112 09:01:26.554826 24181 debug-util.cc:398] Leaking SignalData structure 0x7b0800073600 after lost signal to thread 24053
W20250112 09:01:26.555667 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000865e0 after lost signal to thread 24184
I20250112 09:01:26.930266 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:26.930651 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:01:28.023204 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000bec80 after lost signal to thread 24053
W20250112 09:01:28.024068 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000dd9c0 after lost signal to thread 24184
I20250112 09:01:31.936902 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:31.937326 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:01:32.291514 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000caf80 after lost signal to thread 24197
W20250112 09:01:32.292810 24332 debug-util.cc:398] Leaking SignalData structure 0x7b0800072220 after lost signal to thread 24335
W20250112 09:01:35.772763 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb1e0 after lost signal to thread 23900
W20250112 09:01:35.773689 24037 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca7c0 after lost signal to thread 24040
W20250112 09:01:36.874603 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 116: this message will repeat every 5th retry.
I20250112 09:01:36.942520 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:36.942869 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:37.539932 28980 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:01:37.540504 28980 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:01:37.542526 28980 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:01:37.545120 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:01:37.548262 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:37.548872 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:01:37.549120 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:01:37.549628 28980 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:01:38.826715 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc4c0 after lost signal to thread 24197
W20250112 09:01:38.827546 24332 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1680 after lost signal to thread 24335
I20250112 09:01:39.671473 29024 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:01:39.672012 29024 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:01:39.676097 29024 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:01:39.676785 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:01:39.677515 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:01:39.677868 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:01:39.678485 29024 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:01:41.949092 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:41.949504 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:01:43.802346 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5f80 after lost signal to thread 24660
W20250112 09:01:43.803061 24751 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba9c0 after lost signal to thread 24754
W20250112 09:01:43.828572 24336 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (577 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:01:44.710827 24185 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (462 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:01:44.928431 24042 heartbeater.cc:643] Failed to heartbeat to 127.21.42.253:36201 (485 consecutive failures): Network error: Failed to ping master at 127.21.42.253:36201: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:46.026062 24181 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5e80 after lost signal to thread 24053
W20250112 09:01:46.026944 24181 debug-util.cc:398] Leaking SignalData structure 0x7b0800086b40 after lost signal to thread 24184
I20250112 09:01:46.256965 29144 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:01:46.257498 29144 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:01:46.259140 29144 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:01:46.261478 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:01:46.264132 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:46.264861 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:01:46.265127 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:01:46.265656 29144 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:01:46.307283 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 121: this message will repeat every 5th retry.
I20250112 09:01:46.984376 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:46.984711 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:48.440251 29190 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:01:48.440665 29190 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:01:48.442127 29190 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:01:48.446981 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:01:48.447451 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:01:48.447803 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:01:48.448446 29190 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:01:51.990787 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:51.991233 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:01:55.636363 29333 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:01:55.636902 29333 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:01:55.639293 29333 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:01:55.641041 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:01:55.641484 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:01:55.641824 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:01:55.642421 29333 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 09:01:55.770838 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 126: this message will repeat every 5th retry.
I20250112 09:01:56.996482 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:01:56.996801 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:02.002801 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:02.003279 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:02.065826 29458 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:02:02.066447 29458 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:02:02.070556 29458 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:02:02.071010 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:02:02.071588 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:02:02.071846 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:02:02.072428 29458 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:02:03.115617 29481 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:03.116200 29481 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:03.117750 29481 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:03.120095 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:02:03.123226 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:03.123715 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:03.124032 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:03.124580 29481 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:02:05.562894 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 131: this message will repeat every 5th retry.
I20250112 09:02:07.008415 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:07.008793 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:12.015051 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:12.015517 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:02:15.783844 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 136: this message will repeat every 5th retry.
I20250112 09:02:16.717780 29741 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:02:16.718317 29741 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:02:16.721052 29741 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:02:16.722818 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:02:16.723331 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:02:16.723604 24224 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:02:16.724224 29741 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:02:17.020817 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:17.021152 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:20.604784 29814 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:20.605443 29814 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:20.608022 29814 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:20.610895 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:02:20.614133 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:20.614797 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:20.615057 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:20.615628 29814 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:02:22.026600 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:22.027015 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:22.276068 29845 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:02:22.276674 29845 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:02:22.278537 29845 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:02:22.280067 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:02:22.280552 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:02:22.280822 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:02:22.281374 29845 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:02:23.389616 29868 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:23.390091 29868 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:23.391661 29868 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:23.396819 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:23.397311 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:23.397595 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:23.398128 29868 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 09:02:26.537812 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 141: this message will repeat every 5th retry.
I20250112 09:02:27.032394 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:27.032763 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:30.523829 30010 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:30.524483 30010 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:30.526307 30010 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:30.528797 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 09:02:30.531498 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:30.532258 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:30.532514 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:30.533041 30010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:02:32.038156 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:32.038544 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:02:36.818279 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 146: this message will repeat every 5th retry.
I20250112 09:02:36.964211 30135 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:36.964671 30135 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:36.966166 30135 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:36.968600 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:02:36.971356 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:36.971803 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:36.972102 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:36.972596 30135 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:02:37.043349 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:37.043746 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:41.333369 30216 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:02:41.333998 30216 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:02:41.336915 30216 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:02:41.338017 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:02:41.338447 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:02:41.338747 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:02:41.339319 30216 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 09:02:42.050132 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:42.050664 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:02:44.245376 24336 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (723 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:44.916486 30288 raft_consensus.cc:491] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 09:02:44.917023 30288 raft_consensus.cc:513] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff253b3e7a14f43b118105fda65322b" member_type: VOTER last_known_addr { host: "127.21.42.193" port: 42101 } } peers { permanent_uuid: "7679613902d342678beaffab898dfd7d" member_type: VOTER last_known_addr { host: "127.21.42.195" port: 41233 } } peers { permanent_uuid: "8f4921885fee48319c7132725c7691bf" member_type: VOTER last_known_addr { host: "127.21.42.194" port: 37375 } }
I20250112 09:02:44.918818 30288 leader_election.cc:290] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101), 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375)
W20250112 09:02:44.920405 24224 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ff253b3e7a14f43b118105fda65322b (127.21.42.193:42101): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
W20250112 09:02:44.920866 24222 leader_election.cc:343] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Not found: Tablet not found: 8dfa21f18e8f4c8da4681ec1ef221720
I20250112 09:02:44.921110 24222 leader_election.cc:304] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7679613902d342678beaffab898dfd7d; no voters: 6ff253b3e7a14f43b118105fda65322b, 8f4921885fee48319c7132725c7691bf
I20250112 09:02:44.921623 30288 raft_consensus.cc:2743] T 8dfa21f18e8f4c8da4681ec1ef221720 P 7679613902d342678beaffab898dfd7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 09:02:45.115104 24186 heartbeater.cc:643] Failed to heartbeat to 127.21.42.253:36201 (604 consecutive failures): Network error: Failed to ping master at 127.21.42.253:36201: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:45.399039 24041 heartbeater.cc:643] Failed to heartbeat to 127.21.42.252:36643 (583 consecutive failures): Network error: Failed to ping master at 127.21.42.252:36643: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:47.056218 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:47.056646 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 09:02:47.068818 24222 consensus_peers.cc:487] T 77a9b2f822354182b0e36c55cce850ce P 7679613902d342678beaffab898dfd7d -> Peer 8f4921885fee48319c7132725c7691bf (127.21.42.194:37375): Couldn't send request to peer 8f4921885fee48319c7132725c7691bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77a9b2f822354182b0e36c55cce850ce. This is attempt 151: this message will repeat every 5th retry.
I20250112 09:02:48.069217 30347 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e7c2dbf455f04f4fbcc282c86eae73e1)
I20250112 09:02:48.069764 30347 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bb7660e3b04a9ca5fb697b25b17bbc" member_type: VOTER last_known_addr { host: "127.21.42.254" port: 41739 } } peers { permanent_uuid: "5a6dc008c89249efbae18aa4e34b9ea8" member_type: VOTER last_known_addr { host: "127.21.42.253" port: 36201 } } peers { permanent_uuid: "e7c2dbf455f04f4fbcc282c86eae73e1" member_type: VOTER last_known_addr { host: "127.21.42.252" port: 36643 } }
I20250112 09:02:48.071374 30347 leader_election.cc:290] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201), e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643)
W20250112 09:02:48.074680 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 09:02:48.079352 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5a6dc008c89249efbae18aa4e34b9ea8 (127.21.42.253:36201): Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111)
W20250112 09:02:48.080199 24706 leader_election.cc:336] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e7c2dbf455f04f4fbcc282c86eae73e1 (127.21.42.252:36643): Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
I20250112 09:02:48.080530 24706 leader_election.cc:304] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53bb7660e3b04a9ca5fb697b25b17bbc; no voters: 5a6dc008c89249efbae18aa4e34b9ea8, e7c2dbf455f04f4fbcc282c86eae73e1
I20250112 09:02:48.081071 30347 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 53bb7660e3b04a9ca5fb697b25b17bbc [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 09:02:52.063179 24658 master_runner.cc:186] Error getting master registration for 127.21.42.253:36201: Network error: Client connection negotiation failed: client connection to 127.21.42.253:36201: connect: Connection refused (error 111),
I20250112 09:02:52.063767 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 09:02:52.114826 24709 maintenance_manager.cc:418] P 53bb7660e3b04a9ca5fb697b25b17bbc: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 845 bytes on disk
I20250112 09:02:52.127143 24708 maintenance_manager.cc:642] P 53bb7660e3b04a9ca5fb697b25b17bbc: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"dirs.queue_time_us":757,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":246,"thread_start_us":856,"threads_started":2,"wal-append.queue_time_us":1302}
W20250112 09:02:52.478549 24349 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:02:52.479089 24349 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:02:52.479344 24349 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
W20250112 09:02:52.479661 24349 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
F20250112 09:02:52.492022 24355 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a08e8ce40f9a4a72bfd0c37e4d8b5c8a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250112 09:02:52.493709 24360 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.21.42.254:41739,127.21.42.253:36201,127.21.42.252:36643, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.42.252:36643: connect: Connection refused (error 111)
*** Aborted at 1736672572 (unix time) try "date -d @1736672572" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000054ab) received by PID 21675 (TID 0x7f4ab04fe700) from PID 21675; stack trace: ***
@ 0x2dac70 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2dd153 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f4ac1d8f980 (unknown) at ??:0
@ 0x7f4abcfa8fb7 gsignal at ??:0
@ 0x7f4abcfaa921 abort at ??:0
@ 0x2e0c66 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f4ac0063da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f4ac0064950 google::LogMessage::Flush() at ??:0
@ 0x7f4ac0069a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35cda4 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38abbe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x38aa71 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38a1bb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dab30 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f4ac1d846db start_thread at ??:0
@ 0x7f4abd08b71f clone at ??:0