Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251209 20:55:06.368409 11795 test_util.cc:276] Using random seed: 543333750
2025-12-09T20:55:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T20:55:06Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskd5EcMJ/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskd5EcMJ/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/security.properties
2025-12-09 20:55:12: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskd5EcMJ/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskd5EcMJ/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251209 20:55:19.943022 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--webserver_interface=127.11.132.254
--webserver_port=0
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:20.235781 12036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:20.236052 12036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:20.236171 12036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:20.243880 12036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:20.243950 12036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:20.244019 12036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:20.244102 12036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:20.252883 12036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.132.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:20.254263 12036 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:20.256019 12036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:20.263499 12041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:20.263473 12042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:20.264195 12044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:20.265121 12036 server_base.cc:1047] running on GCE node
I20251209 20:55:20.265774 12036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:20.266930 12036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:20.268113 12036 hybrid_clock.cc:648] HybridClock initialized: now 1765313720268080 us; error 23 us; skew 500 ppm
I20251209 20:55:20.268399 12036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:20.270218 12036 webserver.cc:492] Webserver started at http://127.11.132.254:39627/ using document root <none> and password file <none>
I20251209 20:55:20.270888 12036 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:20.270988 12036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:20.271268 12036 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:20.274173 12036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/instance:
uuid: "ec484016839448df924d26e6175a3176"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.274802 12036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal/instance:
uuid: "ec484016839448df924d26e6175a3176"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.279333 12036 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251209 20:55:20.282397 12050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:20.283504 12036 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 20:55:20.283674 12036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
uuid: "ec484016839448df924d26e6175a3176"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.283864 12036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:20.308769 12036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:20.309478 12036 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:20.309706 12036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:20.328130 12036 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.254:40601
I20251209 20:55:20.328163 12101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.254:40601 every 8 connection(s)
I20251209 20:55:20.329440 12036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
I20251209 20:55:20.333233 12102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:20.337299 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12036
I20251209 20:55:20.337531 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal/instance
I20251209 20:55:20.338490 12102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:20.339987 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--webserver_interface=127.11.132.253
--webserver_port=0
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:20.354503 12102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
W20251209 20:55:20.355594 12052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:20.356956 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:20.406301 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } attempt: 1
W20251209 20:55:20.408069 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:20.476356 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } attempt: 2
W20251209 20:55:20.479298 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:20.570731 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } attempt: 3
W20251209 20:55:20.572590 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:20.652891 12105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:20.653134 12105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:20.653201 12105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:20.660625 12105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:20.660697 12105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:20.660751 12105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:20.660797 12105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:20.669972 12105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.132.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:20.671357 12105 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:20.672962 12105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:20.679925 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:20.679936 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:20.680804 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:20.681049 12105 server_base.cc:1047] running on GCE node
I20251209 20:55:20.681727 12105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:20.682909 12105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:20.684082 12105 hybrid_clock.cc:648] HybridClock initialized: now 1765313720684049 us; error 29 us; skew 500 ppm
I20251209 20:55:20.684365 12105 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:20.686245 12105 webserver.cc:492] Webserver started at http://127.11.132.253:34115/ using document root <none> and password file <none>
I20251209 20:55:20.686920 12105 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:20.687005 12105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:20.687333 12105 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:20.689522 12105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/instance:
uuid: "1cc9184e29d6430ca18a3fe7984c725c"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.690240 12105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal/instance:
uuid: "1cc9184e29d6430ca18a3fe7984c725c"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.694712 12105 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20251209 20:55:20.698447 12120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:20.699847 12105 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251209 20:55:20.700017 12105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
uuid: "1cc9184e29d6430ca18a3fe7984c725c"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:20.700217 12105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:20.715798 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } attempt: 4
W20251209 20:55:20.717473 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:20.721015 12105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:20.721670 12105 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:20.721899 12105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:20.740509 12105 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.253:34647
I20251209 20:55:20.740546 12171 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.253:34647 every 8 connection(s)
I20251209 20:55:20.741875 12105 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
I20251209 20:55:20.745617 12172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:20.746102 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12105
I20251209 20:55:20.746251 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal/instance
I20251209 20:55:20.747287 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=0
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:20.751209 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:20.763777 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:20.771823 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } has no permanent_uuid. Determining permanent_uuid...
W20251209 20:55:20.772682 12122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:20.774039 12172 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:20.823319 12172 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 1
W20251209 20:55:20.825119 12172 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:20.893390 12172 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 2
W20251209 20:55:20.895071 12172 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:20.986368 12172 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 3
W20251209 20:55:20.987977 12172 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:21.016695 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } attempt: 5
I20251209 20:55:21.022631 12102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } has no permanent_uuid. Determining permanent_uuid...
W20251209 20:55:21.024187 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:21.052767 12173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:21.053022 12173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:21.053138 12173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:21.060405 12173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:21.060501 12173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:21.060601 12173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:21.060662 12173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:21.069340 12173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:21.070726 12173 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:21.072377 12173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251209 20:55:21.075376 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 1
W20251209 20:55:21.076906 12102 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:21.079236 12181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:21.079607 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:21.079252 12182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:21.081108 12173 server_base.cc:1047] running on GCE node
I20251209 20:55:21.081707 12173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:21.082818 12173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:21.084043 12173 hybrid_clock.cc:648] HybridClock initialized: now 1765313721083985 us; error 28 us; skew 500 ppm
I20251209 20:55:21.084403 12173 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:21.086184 12173 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:21.086781 12173 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:21.086859 12173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:21.087148 12173 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:21.089198 12173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/instance:
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:21.089870 12173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal/instance:
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:21.094355 12173 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251209 20:55:21.097108 12190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:21.098145 12173 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 20:55:21.098258 12173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:21.098399 12173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:21.114630 12173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:21.115264 12173 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:21.115445 12173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:21.131217 12172 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 4
W20251209 20:55:21.133173 12172 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:21.133267 12173 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:21.133315 12241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:21.134523 12173 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:21.138105 12242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:21.141805 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12173
I20251209 20:55:21.142026 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal/instance
I20251209 20:55:21.144313 12242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:21.145099 12102 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 2
I20251209 20:55:21.167810 12242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:21.168005 12102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap starting.
I20251209 20:55:21.172047 12102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:21.173439 12102 log.cc:826] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:21.174000 12242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } has no permanent_uuid. Determining permanent_uuid...
I20251209 20:55:21.179036 12102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: No bootstrap required, opened a new log
I20251209 20:55:21.184715 12102 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.185001 12242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:21.185015 12102 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:21.185127 12102 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec484016839448df924d26e6175a3176, State: Initialized, Role: FOLLOWER
I20251209 20:55:21.185731 12102 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.187556 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.188001 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.188613 12102 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:21.188697 12242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:21.189924 12242 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:21.192888 12242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: No bootstrap required, opened a new log
I20251209 20:55:21.198223 12242 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.198465 12242 raft_consensus.cc:385] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:21.198580 12242 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:21.199163 12242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.200006 12262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.200277 12262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.200946 12242 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
W20251209 20:55:21.218811 12272 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 20:55:21.218962 12272 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 20:55:21.217161 12036 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: OK, instance_id { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" instance_seqno: 1765313720735375 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251209 20:55:21.228747 12036 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:21.233713 12173 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: OK, instance_id { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" instance_seqno: 1765313720735375 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251209 20:55:21.241856 12173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:55:21.245582 12283 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 20:55:21.245683 12283 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 20:55:21.274479 12262 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:21.274729 12262 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.276484 12262 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:21.279982 12077 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:21.280428 12077 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 0.
I20251209 20:55:21.282567 12191 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters:
I20251209 20:55:21.283141 12262 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:21.283286 12262 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:21.283381 12262 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:21.286324 12262 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.287413 12077 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176"
I20251209 20:55:21.287612 12077 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:21.288141 12262 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 1 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:21.291513 12077 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 1.
I20251209 20:55:21.292034 12191 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters:
I20251209 20:55:21.292480 12262 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Leader election won for term 1
W20251209 20:55:21.312122 12056 tablet.cc:2378] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 20:55:21.321247 12262 raft_consensus.cc:697] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 LEADER]: Becoming Leader. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Running, Role: LEADER
I20251209 20:55:21.324957 12262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.338560 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.338871 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:21.356891 12287 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:21.359936 12287 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20251209 20:55:21.367105 12196 tablet.cc:2378] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 20:55:21.384508 12077 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Refusing update from remote peer 752a47ea6a344edba26b69df6ea3d482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 20:55:21.385543 12285 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:21.413086 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.413348 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.416513 12288 mvcc.cc:204] Tried to move back new op lower bound from 7230725002773811200 to 7230725002555863040. Current Snapshot: MvccSnapshot[applied={T|T < 7230725002773811200}]
I20251209 20:55:21.432368 12172 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } attempt: 5
I20251209 20:55:21.446244 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap starting.
I20251209 20:55:21.454597 12172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:21.456708 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.456949 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:21.458956 12172 log.cc:826] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:21.460974 12287 catalog_manager.cc:1357] Generated new cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:21.461055 12287 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:21.474939 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: No bootstrap required, opened a new log
I20251209 20:55:21.478811 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.479031 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.486671 12172 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.486967 12172 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:21.487075 12172 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9184e29d6430ca18a3fe7984c725c, State: Initialized, Role: FOLLOWER
I20251209 20:55:21.487764 12172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:21.500151 12294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.500535 12294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.501464 12172 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:21.501819 12145 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:21.502417 12146 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c"
I20251209 20:55:21.503316 12147 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c" is_pre_election: true
I20251209 20:55:21.506415 12145 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Refusing update from remote peer 752a47ea6a344edba26b69df6ea3d482: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 20:55:21.507067 12285 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:21.509099 12287 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 20:55:21.510648 12287 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:21.552290 12287 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Generated new TSK 0
I20251209 20:55:21.553354 12287 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:21.554630 12287 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:21.554673 12262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.554909 12262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:21.554775 12294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.554996 12294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.555298 12299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:21.555613 12299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:21.555661 12287 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:21.571897 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.193:0
--local_ip_for_outbound_sockets=127.11.132.193
--webserver_interface=127.11.132.193
--webserver_port=0
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251209 20:55:21.642175 12315 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:21.642283 12315 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: loading cluster ID for follower catalog manager: success
I20251209 20:55:21.645862 12315 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: acquiring CA information for follower catalog manager: success
I20251209 20:55:21.647986 12315 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251209 20:55:22.008333 12313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:22.008603 12313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:22.008746 12313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:22.016098 12313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:22.016266 12313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.193
I20251209 20:55:22.025018 12313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.132.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.193
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:22.026352 12313 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:22.027940 12313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:22.036352 12321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.036888 12322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.037581 12324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:22.038419 12313 server_base.cc:1047] running on GCE node
I20251209 20:55:22.038897 12313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:22.039755 12313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:22.040967 12313 hybrid_clock.cc:648] HybridClock initialized: now 1765313722040916 us; error 42 us; skew 500 ppm
I20251209 20:55:22.041302 12313 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:22.043708 12313 webserver.cc:492] Webserver started at http://127.11.132.193:45233/ using document root <none> and password file <none>
I20251209 20:55:22.044390 12313 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:22.044473 12313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:22.044802 12313 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:22.046993 12313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data/instance:
uuid: "412a1acaab96447b97a475fb3f94fa12"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.047797 12313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/instance:
uuid: "412a1acaab96447b97a475fb3f94fa12"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.051609 12329 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:22.051829 12329 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:22.053120 12329 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:22.053431 12313 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20251209 20:55:22.053771 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:22.053826 12147 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c" is_pre_election: true
I20251209 20:55:22.054636 12052 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:22.055007 12329 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:55:22.057152 12331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.058374 12313 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251209 20:55:22.058526 12313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal
uuid: "412a1acaab96447b97a475fb3f94fa12"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.058724 12313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:22.082898 12313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:22.083748 12313 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:22.083995 12313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:22.084789 12313 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 20:55:22.086180 12313 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 20:55:22.086269 12313 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.086376 12313 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 20:55:22.086439 12313 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.117094 12313 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.193:43197
I20251209 20:55:22.117163 12443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.193:43197 every 8 connection(s)
I20251209 20:55:22.118314 12313 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/data/info.pb
I20251209 20:55:22.119095 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12313
I20251209 20:55:22.119284 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/instance
I20251209 20:55:22.129616 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.194:0
--local_ip_for_outbound_sockets=127.11.132.194
--webserver_interface=127.11.132.194
--webserver_port=0
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251209 20:55:22.140403 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:22.140687 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.141585 12444 heartbeater.cc:507] Master 127.11.132.252:42679 requested a full tablet report, sending...
I20251209 20:55:22.144130 12207 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:22.145474 12207 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.132.193:54109
I20251209 20:55:22.146780 12445 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:22.146907 12445 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.147300 12445 heartbeater.cc:507] Master 127.11.132.254:40601 requested a full tablet report, sending...
I20251209 20:55:22.148850 12066 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:22.156858 12446 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:22.156966 12446 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.157241 12446 heartbeater.cc:507] Master 127.11.132.253:34647 requested a full tablet report, sending...
I20251209 20:55:22.158545 12137 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:22.221086 12272 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:22.221191 12272 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: loading cluster ID for follower catalog manager: success
I20251209 20:55:22.224375 12272 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: acquiring CA information for follower catalog manager: success
I20251209 20:55:22.226378 12272 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251209 20:55:22.455727 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:22.456023 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:22.456151 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:22.463966 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:22.464161 12453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.194
I20251209 20:55:22.472734 12453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.132.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.194
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:22.474236 12453 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:22.475804 12453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:22.484155 12459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.484387 12460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.485426 12462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:22.485903 12453 server_base.cc:1047] running on GCE node
I20251209 20:55:22.486362 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:22.487035 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:22.488214 12453 hybrid_clock.cc:648] HybridClock initialized: now 1765313722488180 us; error 22 us; skew 500 ppm
I20251209 20:55:22.488478 12453 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:22.490438 12453 webserver.cc:492] Webserver started at http://127.11.132.194:38929/ using document root <none> and password file <none>
I20251209 20:55:22.491089 12453 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:22.491194 12453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:22.491515 12453 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:22.493499 12453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data/instance:
uuid: "3cb2248952314571831b6958a0f0a7b4"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.494195 12453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/instance:
uuid: "3cb2248952314571831b6958a0f0a7b4"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.498679 12453 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251209 20:55:22.501653 12468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.502949 12453 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251209 20:55:22.503104 12453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal
uuid: "3cb2248952314571831b6958a0f0a7b4"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.503284 12453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:22.530231 12453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:22.530997 12453 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:22.531231 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:22.532069 12453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 20:55:22.533491 12453 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 20:55:22.533571 12453 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.533638 12453 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 20:55:22.533684 12453 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.543929 12505 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:22.544160 12505 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:22.545527 12505 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:22.546046 12076 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:22.546191 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:22.546887 12122 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:22.547283 12505 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:55:22.568949 12453 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.194:41287
I20251209 20:55:22.569031 12581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.194:41287 every 8 connection(s)
I20251209 20:55:22.570230 12453 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/data/info.pb
I20251209 20:55:22.571597 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12453
I20251209 20:55:22.571782 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/instance
I20251209 20:55:22.576586 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.195:0
--local_ip_for_outbound_sockets=127.11.132.195
--webserver_interface=127.11.132.195
--webserver_port=0
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251209 20:55:22.590262 12583 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:22.590415 12584 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:22.590606 12584 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.590664 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:22.590611 12583 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.590850 12582 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.591667 12584 heartbeater.cc:507] Master 127.11.132.253:34647 requested a full tablet report, sending...
I20251209 20:55:22.591735 12582 heartbeater.cc:507] Master 127.11.132.252:42679 requested a full tablet report, sending...
I20251209 20:55:22.591763 12583 heartbeater.cc:507] Master 127.11.132.254:40601 requested a full tablet report, sending...
I20251209 20:55:22.593235 12066 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:22.593791 12207 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:22.594527 12207 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.132.194:56539
I20251209 20:55:22.595185 12137 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:22.649585 12444 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:22.654065 12590 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:22.654223 12590 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:22.654719 12590 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:22.655179 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:22.655248 12147 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c" is_pre_election: true
I20251209 20:55:22.655738 12052 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:22.655963 12590 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:55:22.886435 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:22.886670 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:22.886790 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:22.893823 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:22.893996 12589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.195
I20251209 20:55:22.902441 12589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.132.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.195
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:22.903913 12589 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:22.905592 12589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:22.914306 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.914892 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:22.915349 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:22.915927 12589 server_base.cc:1047] running on GCE node
I20251209 20:55:22.916361 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:22.917057 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:22.918248 12589 hybrid_clock.cc:648] HybridClock initialized: now 1765313722918218 us; error 27 us; skew 500 ppm
I20251209 20:55:22.918498 12589 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:22.920369 12589 webserver.cc:492] Webserver started at http://127.11.132.195:34939/ using document root <none> and password file <none>
I20251209 20:55:22.921011 12589 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:22.921120 12589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:22.921456 12589 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 20:55:22.923534 12589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data/instance:
uuid: "157c709733f1484799756549aa456c5f"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.924269 12589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/instance:
uuid: "157c709733f1484799756549aa456c5f"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.928488 12589 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251209 20:55:22.931327 12604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.932405 12589 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 20:55:22.932544 12589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal
uuid: "157c709733f1484799756549aa456c5f"
format_stamp: "Formatted at 2025-12-09 20:55:22 on dist-test-slave-nk8j"
I20251209 20:55:22.932725 12589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:22.942871 12589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:22.943550 12589 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:22.943866 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:22.944645 12589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 20:55:22.946012 12589 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 20:55:22.946097 12589 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.946189 12589 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 20:55:22.946255 12589 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:22.976555 12589 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.195:38447
I20251209 20:55:22.976581 12716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.195:38447 every 8 connection(s)
I20251209 20:55:22.977766 12589 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/data/info.pb
I20251209 20:55:22.984612 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12589
I20251209 20:55:22.984779 11795 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/instance
I20251209 20:55:22.993676 12718 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:22.993795 12719 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:22.994056 12718 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.994059 12719 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.994123 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:22.994282 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:22.995105 12717 heartbeater.cc:507] Master 127.11.132.252:42679 requested a full tablet report, sending...
I20251209 20:55:22.995136 12718 heartbeater.cc:507] Master 127.11.132.254:40601 requested a full tablet report, sending...
I20251209 20:55:22.995158 12719 heartbeater.cc:507] Master 127.11.132.253:34647 requested a full tablet report, sending...
I20251209 20:55:22.996469 12207 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:22.996564 12137 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:22.996637 12066 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:22.997417 12207 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.132.195:33273
I20251209 20:55:23.008762 11795 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 20:55:23.036242 12205 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_75ebe9a697e04984b340737cc08cb914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.036441 12206 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_79b3eb661bba41bb99071683748470b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.037281 12204 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.037736 12207 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_c574530850844bf8b58ae3169558d6b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.038480 12203 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_6f837998c1284f3c9172483c60da8813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.038959 12202 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_08951a6a698b49d0b1229aa3bebf4877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.039050 12201 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_25ee0c76d07240769b864a5048b8db94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.039542 12200 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_27e25321226d4880ad3b53e9936fd7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.040217 12199 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.040941 12198 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "default.Table_e2d4d7aea5fd4d41a0b084f0b7726b94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 20:55:23.044791 12206 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_79b3eb661bba41bb99071683748470b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 20:55:23.050616 11795 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu pid 12173
I20251209 20:55:23.051647 12237 generic_service.cc:196] Checking for leaks (request via RPC)
I20251209 20:55:23.096922 12582 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:23.145464 12764 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:23.145830 12764 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:23.147049 12076 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:23.148104 12764 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:23.176407 12765 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:23.176616 12765 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:23.177549 12147 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c" is_pre_election: true
I20251209 20:55:23.177759 12147 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec484016839448df924d26e6175a3176 in term 1.
I20251209 20:55:23.178229 12052 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176; no voters:
I20251209 20:55:23.184715 12765 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:23.185177 12765 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251209 20:55:23.185472 12765 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:23.185722 12765 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 1 FOLLOWER]: Advancing to term 2
I20251209 20:55:23.190402 12765 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:23.191711 12765 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 election: Requested vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:23.193930 12147 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c"
I20251209 20:55:23.194135 12147 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 1 FOLLOWER]: Advancing to term 2
I20251209 20:55:23.196785 12147 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec484016839448df924d26e6175a3176 in term 2.
I20251209 20:55:23.201119 12052 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176; no voters:
I20251209 20:55:23.201730 12766 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Leader election won for term 2
I20251209 20:55:23.201947 12766 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 LEADER]: Becoming Leader. State: Replica: ec484016839448df924d26e6175a3176, State: Running, Role: LEADER
I20251209 20:55:23.202903 12766 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:23.217125 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec484016839448df924d26e6175a3176. Latest consensus state: current_term: 2 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:23.217325 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:23.228070 12768 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:23.229980 12217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:23.230799 12216 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482"
I20251209 20:55:23.231359 12768 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:23.231374 12215 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:23.232338 12768 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:23.232414 12768 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:23.233143 12076 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 LEADER]: Rejecting Update request from peer 752a47ea6a344edba26b69df6ea3d482 for earlier term 1. Current term is 2. Ops: []
I20251209 20:55:23.233839 12768 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:23.234064 11795 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu with pid 12173
I20251209 20:55:23.235292 12768 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Loaded TSK: 0
I20251209 20:55:23.236399 12768 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:23.237411 12768 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:23.238336 12768 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
W20251209 20:55:23.260174 12725 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:23.263854 12470 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:23.264289 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.11.132.252:42679 (error 108)
W20251209 20:55:23.265430 12582 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251209 20:55:23.269258 12065 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_27e25321226d4880ad3b53e9936fd7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.269908 12067 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.269371 12063 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_79b3eb661bba41bb99071683748470b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.270704 12061 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_75ebe9a697e04984b340737cc08cb914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.271144 12066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_e2d4d7aea5fd4d41a0b084f0b7726b94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.269990 12059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_6f837998c1284f3c9172483c60da8813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.272297 12060 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.271785 12062 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_c574530850844bf8b58ae3169558d6b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.271962 12064 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_25ee0c76d07240769b864a5048b8db94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 20:55:23.274011 12728 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 20:55:23.261820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.132.252:42679 (local address 127.0.0.1:38836)
1209 20:55:23.262146 (+ 326us) negotiation.cc:107] Waiting for socket to connect
1209 20:55:23.262166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1209 20:55:23.262327 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 20:55:23.272447 (+ 10120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86}
I20251209 20:55:23.274403 12058 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_08951a6a698b49d0b1229aa3bebf4877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 20:55:23.275537 12122 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:23.275688 12122 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:23.277480 12764 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:55:23.278280 12052 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:23.281075 12065 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_27e25321226d4880ad3b53e9936fd7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.281175 12067 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_540977ad57d64d6cb18074b1dcbc41ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.283977 12061 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_75ebe9a697e04984b340737cc08cb914 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.284942 12066 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.285784 12059 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6f837998c1284f3c9172483c60da8813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.286532 12060 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.287420 12062 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c574530850844bf8b58ae3169558d6b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.288007 12058 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_08951a6a698b49d0b1229aa3bebf4877 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.288826 12064 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_25ee0c76d07240769b864a5048b8db94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251209 20:55:23.289515 12063 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_79b3eb661bba41bb99071683748470b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 20:55:23.498172 12067 catalog_manager.cc:2174] created HMS catalog entry for table [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:23.499126 12717 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:23.506834 12147 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Refusing update from remote peer ec484016839448df924d26e6175a3176: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20251209 20:55:23.507345 12052 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 -> Peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Couldn't send request to peer 752a47ea6a344edba26b69df6ea3d482. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251209 20:55:23.507404 12717 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:23.507755 12765 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251209 20:55:23.524425 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec484016839448df924d26e6175a3176. Latest consensus state: current_term: 2 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:23.524636 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:23.527164 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:23.527364 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:23.528790 12776 mvcc.cc:204] Tried to move back new op lower bound from 7230725011474841600 to 7230725010247274496. Current Snapshot: MvccSnapshot[applied={T|T < 7230725011474841600}]
I20251209 20:55:23.530987 12775 mvcc.cc:204] Tried to move back new op lower bound from 7230725011474841600 to 7230725010247274496. Current Snapshot: MvccSnapshot[applied={T|T < 7230725011474841600 or (T in {7230725011474841600})}]
I20251209 20:55:23.533911 12067 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef" } new_table_name: "default.Table_5d01adbefb7248b1a99539733a5c4687" modify_external_catalogs: true
I20251209 20:55:23.536433 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:23.536653 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:23.541090 12065 catalog_manager.cc:2174] created HMS catalog entry for table [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:23.554531 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.556034 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06dc5d4749214b5db59251d54dd2b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.578981 12785 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.582343 12785 tablet_bootstrap.cc:654] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.583725 12785 log.cc:826] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:23.585389 12065 master_service.cc:954] ReplaceTablet: received request to replace tablet 06dc5d4749214b5db59251d54dd2b0d9 from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:23.586102 12785 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.586357 12785 ts_tablet_manager.cc:1403] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20251209 20:55:23.590360 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.592962 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06dc5d4749214b5db59251d54dd2b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.592640 12379 tablet_service.cc:1505] Processing CreateTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.593084 12785 raft_consensus.cc:359] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.593444 12785 raft_consensus.cc:385] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.593555 12785 raft_consensus.cc:740] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.593899 12379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06dc5d4749214b5db59251d54dd2b0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.594273 12785 consensus_queue.cc:260] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.603891 12785 ts_tablet_manager.cc:1434] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20251209 20:55:23.605305 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 0f187335da3a4b39b4c453eb470e99e8 (DEFAULT_TABLE table=default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.605738 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f187335da3a4b39b4c453eb470e99e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.605906 12718 heartbeater.cc:499] Master 127.11.132.254:40601 was elected leader, sending a full tablet report...
I20251209 20:55:23.612655 12516 tablet_service.cc:1505] Processing CreateTablet for tablet 0f187335da3a4b39b4c453eb470e99e8 (DEFAULT_TABLE table=default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.612916 12785 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.613147 12516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f187335da3a4b39b4c453eb470e99e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.613461 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 0f187335da3a4b39b4c453eb470e99e8 (DEFAULT_TABLE table=default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.613962 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f187335da3a4b39b4c453eb470e99e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.622462 12785 tablet_bootstrap.cc:654] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.623734 12788 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.623764 12065 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Sending DeleteTablet for 3 replicas of tablet 06dc5d4749214b5db59251d54dd2b0d9
I20251209 20:55:23.624977 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:54366
I20251209 20:55:23.625583 12065 catalog_manager.cc:6724] ReplaceTablet: tablet 06dc5d4749214b5db59251d54dd2b0d9 deleted and replaced by tablet c30b153a590a4fc488749b55b2e6ed29
I20251209 20:55:23.628687 12788 tablet_bootstrap.cc:654] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.629536 12785 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.629683 12785 ts_tablet_manager.cc:1403] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20251209 20:55:23.630213 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:57366
I20251209 20:55:23.630656 12785 raft_consensus.cc:359] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.630890 12785 raft_consensus.cc:385] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.631002 12785 raft_consensus.cc:740] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.631922 12061 catalog_manager.cc:2174] created HMS catalog entry for table [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:23.632365 12789 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.633005 12065 master_service.cc:954] ReplaceTablet: received request to replace tablet 06dc5d4749214b5db59251d54dd2b0d9 from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:23.636569 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:51470
I20251209 20:55:23.637674 12785 consensus_queue.cc:260] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.639065 12065 master_service.cc:954] ReplaceTablet: received request to replace tablet 06dc5d4749214b5db59251d54dd2b0d9 from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:23.641950 12785 ts_tablet_manager.cc:1434] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251209 20:55:23.643154 12061 master_service.cc:954] ReplaceTablet: received request to replace tablet 06dc5d4749214b5db59251d54dd2b0d9 from {username='slave'} at 127.0.0.1:53374
W20251209 20:55:23.643778 12051 catalog_manager.cc:4965] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): delete failed for tablet 06dc5d4749214b5db59251d54dd2b0d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06dc5d4749214b5db59251d54dd2b0d9 already in progress: creating tablet
I20251209 20:55:23.644883 12377 tablet_service.cc:1505] Processing CreateTablet for tablet c30b153a590a4fc488749b55b2e6ed29 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.644977 12651 tablet_service.cc:1505] Processing CreateTablet for tablet c30b153a590a4fc488749b55b2e6ed29 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.645041 12517 tablet_service.cc:1505] Processing CreateTablet for tablet c30b153a590a4fc488749b55b2e6ed29 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.645452 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30b153a590a4fc488749b55b2e6ed29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.645458 12651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30b153a590a4fc488749b55b2e6ed29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.645467 12377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30b153a590a4fc488749b55b2e6ed29. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.648324 12065 master_service.cc:954] ReplaceTablet: received request to replace tablet 06dc5d4749214b5db59251d54dd2b0d9 from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:23.654088 12785 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.658819 12791 tablet_replica.cc:333] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:23.659483 12791 raft_consensus.cc:2243] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:23.659596 12785 tablet_bootstrap.cc:654] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.659934 12791 raft_consensus.cc:2272] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:23.661506 12791 ts_tablet_manager.cc:1916] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:23.661907 12789 tablet_bootstrap.cc:654] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.661962 12788 log.cc:826] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Log is configured to *not* fsync() on all Append() calls
W20251209 20:55:23.666862 12052 catalog_manager.cc:4965] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): delete failed for tablet 06dc5d4749214b5db59251d54dd2b0d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06dc5d4749214b5db59251d54dd2b0d9 already in progress: creating tablet
I20251209 20:55:23.667721 12791 ts_tablet_manager.cc:1929] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:23.667857 12791 log.cc:1199] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/06dc5d4749214b5db59251d54dd2b0d9
I20251209 20:55:23.667958 12785 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.668116 12785 ts_tablet_manager.cc:1403] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20251209 20:55:23.668253 12791 ts_tablet_manager.cc:1950] T 06dc5d4749214b5db59251d54dd2b0d9 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:23.668905 12785 raft_consensus.cc:359] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.669101 12785 raft_consensus.cc:385] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.669201 12785 raft_consensus.cc:740] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.670014 12789 log.cc:826] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:23.670292 12054 catalog_manager.cc:4985] TS 157c709733f1484799756549aa456c5f (127.11.132.195:38447): tablet 06dc5d4749214b5db59251d54dd2b0d9 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]) successfully deleted
I20251209 20:55:23.670651 12785 consensus_queue.cc:260] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.671897 12785 ts_tablet_manager.cc:1434] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251209 20:55:23.678325 12065 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_27e25321226d4880ad3b53e9936fd7a8" } new_table_name: "default.Table_855e1033632d4d57b523f0e18d938547" modify_external_catalogs: true
I20251209 20:55:23.682544 12788 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.685880 12061 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_75ebe9a697e04984b340737cc08cb914" } new_table_name: "default.Table_8f50250b772546b68acb72dc72554936" modify_external_catalogs: true
I20251209 20:55:23.686440 12652 tablet_service.cc:1505] Processing CreateTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.686756 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.686623 12517 tablet_service.cc:1505] Processing CreateTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.687785 12378 tablet_service.cc:1505] Processing CreateTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.688153 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.688163 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.692405 12785 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.693876 12785 tablet_bootstrap.cc:654] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.694366 12788 ts_tablet_manager.cc:1403] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.003s
I20251209 20:55:23.701886 12789 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.702126 12789 ts_tablet_manager.cc:1403] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.006s
I20251209 20:55:23.701855 12788 raft_consensus.cc:359] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.702361 12788 raft_consensus.cc:385] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.702504 12788 raft_consensus.cc:740] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.703294 12788 consensus_queue.cc:260] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.705366 12785 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.705518 12785 ts_tablet_manager.cc:1403] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20251209 20:55:23.706378 12785 raft_consensus.cc:359] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.706549 12785 raft_consensus.cc:385] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.706555 12066 catalog_manager.cc:2174] created HMS catalog entry for table [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:23.706633 12785 raft_consensus.cc:740] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.706875 12785 consensus_queue.cc:260] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.709816 12785 ts_tablet_manager.cc:1434] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251209 20:55:23.718156 12789 raft_consensus.cc:359] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.718564 12789 raft_consensus.cc:385] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.718680 12789 raft_consensus.cc:740] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.719465 12789 consensus_queue.cc:260] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.721726 12583 heartbeater.cc:499] Master 127.11.132.254:40601 was elected leader, sending a full tablet report...
I20251209 20:55:23.722996 12066 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_e2d4d7aea5fd4d41a0b084f0b7726b94" } new_table_name: "default.Table_a55cc08bf00c49778a226eba00e323f2" modify_external_catalogs: true
I20251209 20:55:23.723841 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.724280 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db0bbab30c24ce5befbdc1653251f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.724431 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.724819 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db0bbab30c24ce5befbdc1653251f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.728542 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.728941 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db0bbab30c24ce5befbdc1653251f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.730836 12785 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Bootstrap starting.
W20251209 20:55:23.730943 12720 tablet.cc:2378] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 20:55:23.732529 12785 tablet_bootstrap.cc:654] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.741497 12788 ts_tablet_manager.cc:1434] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.047s user 0.011s sys 0.000s
I20251209 20:55:23.742000 12788 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.744185 12788 tablet_bootstrap.cc:654] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.745663 12785 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.745867 12785 ts_tablet_manager.cc:1403] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20251209 20:55:23.746754 12785 raft_consensus.cc:359] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.746994 12785 raft_consensus.cc:385] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.747087 12785 raft_consensus.cc:740] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.747386 12785 consensus_queue.cc:260] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.751946 12785 ts_tablet_manager.cc:1434] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251209 20:55:23.756146 12789 ts_tablet_manager.cc:1434] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.054s user 0.017s sys 0.000s
I20251209 20:55:23.757969 12059 catalog_manager.cc:2174] created HMS catalog entry for table [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:23.760737 12445 heartbeater.cc:499] Master 127.11.132.254:40601 was elected leader, sending a full tablet report...
I20251209 20:55:23.762814 12789 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.765179 12789 tablet_bootstrap.cc:654] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.769857 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:57366
I20251209 20:55:23.770232 12792 tablet_replica.cc:333] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:23.770745 12792 raft_consensus.cc:2243] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:23.771207 12792 raft_consensus.cc:2272] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:23.773240 12792 ts_tablet_manager.cc:1916] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:23.779663 12792 ts_tablet_manager.cc:1929] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:23.779872 12792 log.cc:1199] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/06dc5d4749214b5db59251d54dd2b0d9
I20251209 20:55:23.780335 12792 ts_tablet_manager.cc:1950] T 06dc5d4749214b5db59251d54dd2b0d9 P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:23.780301 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.780310 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.780682 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2925610d5e4948a0c2d93dd5072a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.780737 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2925610d5e4948a0c2d93dd5072a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.782032 12051 catalog_manager.cc:4985] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): tablet 06dc5d4749214b5db59251d54dd2b0d9 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]) successfully deleted
I20251209 20:55:23.784379 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.784904 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2925610d5e4948a0c2d93dd5072a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.786926 12785 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.788872 12785 tablet_bootstrap.cc:654] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.790725 12785 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.790767 12789 tablet_bootstrap.cc:492] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.790926 12785 ts_tablet_manager.cc:1403] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251209 20:55:23.790925 12789 ts_tablet_manager.cc:1403] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.003s
I20251209 20:55:23.791731 12785 raft_consensus.cc:359] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.792004 12785 raft_consensus.cc:385] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.791965 12789 raft_consensus.cc:359] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.792163 12789 raft_consensus.cc:385] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.792197 12785 raft_consensus.cc:740] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.792261 12789 raft_consensus.cc:740] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.792511 12785 consensus_queue.cc:260] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.793048 12785 ts_tablet_manager.cc:1434] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251209 20:55:23.798669 12789 consensus_queue.cc:260] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } }
I20251209 20:55:23.798955 12379 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:51470
W20251209 20:55:23.799695 12052 catalog_manager.cc:4965] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): delete failed for tablet 06dc5d4749214b5db59251d54dd2b0d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06dc5d4749214b5db59251d54dd2b0d9 already in progress: creating tablet
I20251209 20:55:23.800654 12789 ts_tablet_manager.cc:1434] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20251209 20:55:23.801805 12789 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.803119 12516 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:57366
W20251209 20:55:23.803690 12051 catalog_manager.cc:4952] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): delete failed for tablet 06dc5d4749214b5db59251d54dd2b0d9 because the tablet was not found. No further retry: Not found: Tablet not found: 06dc5d4749214b5db59251d54dd2b0d9
I20251209 20:55:23.804919 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 06dc5d4749214b5db59251d54dd2b0d9 with delete_type TABLET_DATA_DELETED (replaced by tablet c30b153a590a4fc488749b55b2e6ed29) from {username='slave'} at 127.0.0.1:51470
I20251209 20:55:23.805281 12793 tablet_replica.cc:333] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:23.805791 12793 raft_consensus.cc:2243] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:23.806201 12793 raft_consensus.cc:2272] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:23.806232 12059 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_1b5967177d41405ababac4e174a63fc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.807158 12788 tablet_bootstrap.cc:492] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.807305 12788 ts_tablet_manager.cc:1403] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.007s
I20251209 20:55:23.807798 12793 ts_tablet_manager.cc:1916] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:23.808122 12788 raft_consensus.cc:359] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.808312 12788 raft_consensus.cc:385] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.808415 12788 raft_consensus.cc:740] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.808667 12788 consensus_queue.cc:260] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.809165 12788 ts_tablet_manager.cc:1434] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251209 20:55:23.811604 12789 tablet_bootstrap.cc:654] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.813707 12788 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.814528 12793 ts_tablet_manager.cc:1929] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:23.814675 12793 log.cc:1199] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/06dc5d4749214b5db59251d54dd2b0d9
I20251209 20:55:23.815062 12793 ts_tablet_manager.cc:1950] T 06dc5d4749214b5db59251d54dd2b0d9 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:23.815946 12788 tablet_bootstrap.cc:654] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.816602 12052 catalog_manager.cc:4985] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): tablet 06dc5d4749214b5db59251d54dd2b0d9 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]) successfully deleted
I20251209 20:55:23.817481 12789 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.817642 12789 ts_tablet_manager.cc:1403] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20251209 20:55:23.817873 12788 tablet_bootstrap.cc:492] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.818020 12788 ts_tablet_manager.cc:1403] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251209 20:55:23.818380 12789 raft_consensus.cc:359] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.818563 12789 raft_consensus.cc:385] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.818650 12789 raft_consensus.cc:740] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.818825 12788 raft_consensus.cc:359] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.818902 12789 consensus_queue.cc:260] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.819029 12788 raft_consensus.cc:385] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.819375 12789 ts_tablet_manager.cc:1434] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251209 20:55:23.819674 12789 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.820204 12060 catalog_manager.cc:2174] created HMS catalog entry for table [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:23.823040 12788 raft_consensus.cc:740] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.823318 12788 consensus_queue.cc:260] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.826295 12788 ts_tablet_manager.cc:1434] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251209 20:55:23.828050 12585 tablet.cc:2378] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 20:55:23.829023 12060 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_6f837998c1284f3c9172483c60da8813" } new_table_name: "default.Table_581f8285cf154582b0b9bb5096b8ae7a" modify_external_catalogs: true
I20251209 20:55:23.830112 12788 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.831720 12788 tablet_bootstrap.cc:654] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.840849 12788 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.841033 12788 ts_tablet_manager.cc:1403] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20251209 20:55:23.841825 12788 raft_consensus.cc:359] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.842010 12788 raft_consensus.cc:385] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.842105 12788 raft_consensus.cc:740] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.842339 12788 consensus_queue.cc:260] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.844841 12788 ts_tablet_manager.cc:1434] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251209 20:55:23.847945 12788 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.848755 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 0e200643b18b43dd951a9af667bf1dca (DEFAULT_TABLE table=default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.848744 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 0e200643b18b43dd951a9af667bf1dca (DEFAULT_TABLE table=default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.849135 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e200643b18b43dd951a9af667bf1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.849141 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e200643b18b43dd951a9af667bf1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.852273 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 0e200643b18b43dd951a9af667bf1dca (DEFAULT_TABLE table=default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.852658 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e200643b18b43dd951a9af667bf1dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.854640 12785 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.856412 12785 tablet_bootstrap.cc:654] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.861838 12788 tablet_bootstrap.cc:654] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.862262 12062 catalog_manager.cc:2174] created HMS catalog entry for table [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:23.865371 12789 tablet_bootstrap.cc:654] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.867362 12788 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.867569 12788 ts_tablet_manager.cc:1403] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20251209 20:55:23.868424 12788 raft_consensus.cc:359] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.868628 12788 raft_consensus.cc:385] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.868709 12788 raft_consensus.cc:740] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.868748 12785 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.869524 12785 ts_tablet_manager.cc:1403] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20251209 20:55:23.869534 12788 consensus_queue.cc:260] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.870576 12788 ts_tablet_manager.cc:1434] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20251209 20:55:23.870913 12788 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.874749 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.875202 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.876024 12789 tablet_bootstrap.cc:492] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.876187 12789 ts_tablet_manager.cc:1403] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.004s
I20251209 20:55:23.876417 12062 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_a4904681b195452fb7adf16df235f461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:23.876762 12785 raft_consensus.cc:359] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.876986 12785 raft_consensus.cc:385] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.877067 12785 raft_consensus.cc:740] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.877033 12789 raft_consensus.cc:359] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.877233 12789 raft_consensus.cc:385] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.877338 12789 raft_consensus.cc:740] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.877568 12789 consensus_queue.cc:260] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.877667 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.877694 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.878041 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.878062 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.878132 12789 ts_tablet_manager.cc:1434] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251209 20:55:23.878990 12789 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.880584 12789 tablet_bootstrap.cc:654] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
W20251209 20:55:23.883947 12451 tablet.cc:2378] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 20:55:23.877355 12785 consensus_queue.cc:260] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.887132 12785 ts_tablet_manager.cc:1434] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251209 20:55:23.891232 12785 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.893297 12785 tablet_bootstrap.cc:654] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.894169 12789 tablet_bootstrap.cc:492] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.894330 12789 ts_tablet_manager.cc:1403] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20251209 20:55:23.895068 12789 raft_consensus.cc:359] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.895256 12789 raft_consensus.cc:385] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.895344 12789 raft_consensus.cc:740] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.895378 12788 tablet_bootstrap.cc:654] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.895638 12789 consensus_queue.cc:260] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.898922 12789 ts_tablet_manager.cc:1434] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251209 20:55:23.902072 12789 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.904012 12789 tablet_bootstrap.cc:654] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.906693 12785 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.906862 12785 ts_tablet_manager.cc:1403] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20251209 20:55:23.907639 12785 raft_consensus.cc:359] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.907793 12785 raft_consensus.cc:385] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.907967 12789 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.907893 12785 raft_consensus.cc:740] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.908083 12789 ts_tablet_manager.cc:1403] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251209 20:55:23.908187 12785 consensus_queue.cc:260] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.908700 12789 raft_consensus.cc:359] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.908862 12789 raft_consensus.cc:385] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.908944 12789 raft_consensus.cc:740] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.909122 12789 consensus_queue.cc:260] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.911042 12785 ts_tablet_manager.cc:1434] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251209 20:55:23.911716 12058 catalog_manager.cc:2174] created HMS catalog entry for table [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:23.911870 12789 ts_tablet_manager.cc:1434] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251209 20:55:23.913885 12789 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.916291 12788 tablet_bootstrap.cc:492] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.916393 12788 ts_tablet_manager.cc:1403] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.004s
I20251209 20:55:23.916882 12788 raft_consensus.cc:359] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.917058 12788 raft_consensus.cc:385] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.917155 12788 raft_consensus.cc:740] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.917356 12788 consensus_queue.cc:260] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.917812 12788 ts_tablet_manager.cc:1434] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 20:55:23.918077 12788 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.919700 12788 tablet_bootstrap.cc:654] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.920768 12789 tablet_bootstrap.cc:654] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.926647 12789 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:23.926787 12789 ts_tablet_manager.cc:1403] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20251209 20:55:23.927063 12788 tablet_bootstrap.cc:492] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:23.927201 12788 ts_tablet_manager.cc:1403] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20251209 20:55:23.927484 12789 raft_consensus.cc:359] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.927698 12789 raft_consensus.cc:385] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.927778 12789 raft_consensus.cc:740] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.928195 12802 raft_consensus.cc:493] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:23.928146 12788 raft_consensus.cc:359] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.928341 12788 raft_consensus.cc:385] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.928417 12788 raft_consensus.cc:740] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.928418 12802 raft_consensus.cc:515] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.928654 12788 consensus_queue.cc:260] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.930477 12802 leader_election.cc:290] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:23.930598 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=42787
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:23.928025 12789 consensus_queue.cc:260] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.931733 12788 ts_tablet_manager.cc:1434] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251209 20:55:23.932147 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 19c5cdf7ec11442e99363cb062365d38 (DEFAULT_TABLE table=default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.932551 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c5cdf7ec11442e99363cb062365d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.933713 12788 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:23.935164 12789 ts_tablet_manager.cc:1434] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251209 20:55:23.935415 12788 tablet_bootstrap.cc:654] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.937306 12789 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:23.937669 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 19c5cdf7ec11442e99363cb062365d38 (DEFAULT_TABLE table=default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.938016 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c5cdf7ec11442e99363cb062365d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.938868 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c30b153a590a4fc488749b55b2e6ed29" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:23.939250 12672 raft_consensus.cc:2468] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:23.939349 12789 tablet_bootstrap.cc:654] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.939965 12335 leader_election.cc:304] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:23.940356 12802 raft_consensus.cc:2804] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:23.940500 12802 raft_consensus.cc:493] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:23.940583 12802 raft_consensus.cc:3060] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:23.943604 12802 raft_consensus.cc:515] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.944442 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 19c5cdf7ec11442e99363cb062365d38 (DEFAULT_TABLE table=default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:23.944787 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c5cdf7ec11442e99363cb062365d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:23.945885 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c30b153a590a4fc488749b55b2e6ed29" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:23.946034 12672 raft_consensus.cc:3060] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:23.948946 12672 raft_consensus.cc:2468] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:23.949124 12802 leader_election.cc:290] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Requested vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:23.949541 12335 leader_election.cc:304] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:23.950017 12802 raft_consensus.cc:2804] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:23.957690 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c30b153a590a4fc488749b55b2e6ed29" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:23.958220 12537 raft_consensus.cc:2468] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:23.958647 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c30b153a590a4fc488749b55b2e6ed29" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:23.958910 12536 raft_consensus.cc:3060] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:23.963080 12536 raft_consensus.cc:2468] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:23.963528 12785 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:23.964864 12802 raft_consensus.cc:697] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Becoming Leader. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Running, Role: LEADER
I20251209 20:55:23.965375 12802 consensus_queue.cc:237] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.965734 12785 tablet_bootstrap.cc:654] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:23.967445 12058 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:23.967866 12058 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:23.973256 12785 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:23.973456 12785 ts_tablet_manager.cc:1403] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20251209 20:55:23.974442 12785 raft_consensus.cc:359] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.974714 12785 raft_consensus.cc:385] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:23.974807 12785 raft_consensus.cc:740] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:23.975136 12785 consensus_queue.cc:260] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.976809 12785 ts_tablet_manager.cc:1434] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251209 20:55:23.982697 12787 raft_consensus.cc:493] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:23.982890 12787 raft_consensus.cc:515] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.985061 12787 leader_election.cc:290] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:23.986059 12800 raft_consensus.cc:493] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:23.986263 12800 raft_consensus.cc:515] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:23.993129 12800 leader_election.cc:290] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.003032 12064 catalog_manager.cc:2174] created HMS catalog entry for table [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:24.005888 12787 raft_consensus.cc:493] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.006070 12787 raft_consensus.cc:515] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.006685 12787 leader_election.cc:290] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:24.017678 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.018157 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0546be0c64a6287404fd0f3a9d427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.024466 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.024909 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0546be0c64a6287404fd0f3a9d427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.026345 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.026712 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f0546be0c64a6287404fd0f3a9d427. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.028353 12064 master_service.cc:954] ReplaceTablet: received request to replace tablet 4db0bbab30c24ce5befbdc1653251f3a from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:24.032624 12785 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:24.034173 12785 tablet_bootstrap.cc:654] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.037390 12064 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Sending DeleteTablet for 3 replicas of tablet 4db0bbab30c24ce5befbdc1653251f3a
I20251209 20:55:24.037586 12788 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.037782 12788 ts_tablet_manager.cc:1403] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.020s
I20251209 20:55:24.040596 12516 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a with delete_type TABLET_DATA_DELETED (replaced by tablet 73119767050448079a74b823ea6587f5) from {username='slave'} at 127.0.0.1:57366
I20251209 20:55:24.040962 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.041213 12536 raft_consensus.cc:2468] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.038786 12788 raft_consensus.cc:359] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.041517 12788 raft_consensus.cc:385] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.041580 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4db0bbab30c24ce5befbdc1653251f3a" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.041653 12788 raft_consensus.cc:740] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.041777 12537 raft_consensus.cc:2468] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.042075 12792 tablet_replica.cc:333] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:24.042337 12792 raft_consensus.cc:2243] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.042526 12792 raft_consensus.cc:2272] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.043308 12792 ts_tablet_manager.cc:1916] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.043376 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.043579 12785 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:24.043607 12672 raft_consensus.cc:2468] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.043737 12785 ts_tablet_manager.cc:1403] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20251209 20:55:24.044206 12379 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a with delete_type TABLET_DATA_DELETED (replaced by tablet 73119767050448079a74b823ea6587f5) from {username='slave'} at 127.0.0.1:51470
I20251209 20:55:24.044680 12785 raft_consensus.cc:359] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.044858 12785 raft_consensus.cc:385] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.044974 12785 raft_consensus.cc:740] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.044189 12472 leader_election.cc:304] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.045217 12785 consensus_queue.cc:260] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.045509 12800 raft_consensus.cc:2804] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.045612 12800 raft_consensus.cc:493] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.045697 12800 raft_consensus.cc:3060] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.048166 12785 ts_tablet_manager.cc:1434] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251209 20:55:24.048262 12800 raft_consensus.cc:515] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.049023 12064 catalog_manager.cc:6724] ReplaceTablet: tablet 4db0bbab30c24ce5befbdc1653251f3a deleted and replaced by tablet 73119767050448079a74b823ea6587f5
I20251209 20:55:24.049168 12792 ts_tablet_manager.cc:1929] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.049300 12792 log.cc:1199] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/4db0bbab30c24ce5befbdc1653251f3a
I20251209 20:55:24.049561 12800 leader_election.cc:290] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.049706 12792 ts_tablet_manager.cc:1950] T 4db0bbab30c24ce5befbdc1653251f3a P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:24.050426 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet 4db0bbab30c24ce5befbdc1653251f3a with delete_type TABLET_DATA_DELETED (replaced by tablet 73119767050448079a74b823ea6587f5) from {username='slave'} at 127.0.0.1:54366
I20251209 20:55:24.050645 12793 tablet_replica.cc:333] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:24.050757 12605 leader_election.cc:304] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.050853 12793 raft_consensus.cc:2243] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.050964 12791 tablet_replica.cc:333] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:24.051039 12793 raft_consensus.cc:2272] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.051128 12787 raft_consensus.cc:2804] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.051167 12791 raft_consensus.cc:2243] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.051232 12787 raft_consensus.cc:493] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.051316 12791 raft_consensus.cc:2272] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.051337 12787 raft_consensus.cc:3060] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.051903 12793 ts_tablet_manager.cc:1916] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.051920 12791 ts_tablet_manager.cc:1916] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.053225 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.054252 12787 raft_consensus.cc:515] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.055081 12787 leader_election.cc:290] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.055204 12379 tablet_service.cc:1505] Processing CreateTablet for tablet 73119767050448079a74b823ea6587f5 (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.055608 12379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73119767050448079a74b823ea6587f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.055527 12605 leader_election.cc:304] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
W20251209 20:55:24.055789 12605 raft_consensus.cc:2686] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251209 20:55:24.055814 12672 raft_consensus.cc:2393] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3cb2248952314571831b6958a0f0a7b4 in current term 1: Already voted for candidate 157c709733f1484799756549aa456c5f in this term.
I20251209 20:55:24.056285 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.056550 12537 raft_consensus.cc:2393] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 157c709733f1484799756549aa456c5f in current term 1: Already voted for candidate 3cb2248952314571831b6958a0f0a7b4 in this term.
I20251209 20:55:24.057420 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 73119767050448079a74b823ea6587f5 (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.057778 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73119767050448079a74b823ea6587f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.057839 12793 ts_tablet_manager.cc:1929] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.057966 12793 log.cc:1199] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/4db0bbab30c24ce5befbdc1653251f3a
I20251209 20:55:24.058292 12793 ts_tablet_manager.cc:1950] T 4db0bbab30c24ce5befbdc1653251f3a P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:24.051405 12051 catalog_manager.cc:4985] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): tablet 4db0bbab30c24ce5befbdc1653251f3a (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]) successfully deleted
I20251209 20:55:24.059856 12791 ts_tablet_manager.cc:1929] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.060042 12791 log.cc:1199] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/4db0bbab30c24ce5befbdc1653251f3a
I20251209 20:55:24.060225 12516 tablet_service.cc:1505] Processing CreateTablet for tablet 73119767050448079a74b823ea6587f5 (DEFAULT_TABLE table=default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.060443 12791 ts_tablet_manager.cc:1950] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:24.060563 12516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73119767050448079a74b823ea6587f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.061650 12788 consensus_queue.cc:260] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.063773 12054 catalog_manager.cc:4985] TS 157c709733f1484799756549aa456c5f (127.11.132.195:38447): tablet 4db0bbab30c24ce5befbdc1653251f3a (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]) successfully deleted
I20251209 20:55:24.066232 12788 ts_tablet_manager.cc:1434] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.028s user 0.002s sys 0.001s
I20251209 20:55:24.066895 12052 catalog_manager.cc:4985] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): tablet 4db0bbab30c24ce5befbdc1653251f3a (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]) successfully deleted
I20251209 20:55:24.068452 12788 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:24.069896 12785 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:24.070897 12788 tablet_bootstrap.cc:654] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.071513 12785 tablet_bootstrap.cc:654] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.076092 12064 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a2925610d5e4948a0c2d93dd5072a2d from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:24.077560 12789 tablet_bootstrap.cc:492] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.077704 12789 ts_tablet_manager.cc:1403] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.141s user 0.052s sys 0.055s
I20251209 20:55:24.077769 12788 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.077950 12788 ts_tablet_manager.cc:1403] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251209 20:55:24.078631 12789 raft_consensus.cc:359] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.078810 12789 raft_consensus.cc:385] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.078794 12788 raft_consensus.cc:359] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.078929 12789 raft_consensus.cc:740] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.079008 12788 raft_consensus.cc:385] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.079092 12788 raft_consensus.cc:740] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.079341 12788 consensus_queue.cc:260] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.079859 12788 ts_tablet_manager.cc:1434] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251209 20:55:24.080173 12788 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:24.081718 12788 tablet_bootstrap.cc:654] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.081825 12789 consensus_queue.cc:260] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.082549 12789 ts_tablet_manager.cc:1434] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251209 20:55:24.082791 12789 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:24.084926 12785 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:24.085084 12785 ts_tablet_manager.cc:1403] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251209 20:55:24.085855 12785 raft_consensus.cc:359] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.086053 12785 raft_consensus.cc:385] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.086133 12785 raft_consensus.cc:740] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.086385 12785 consensus_queue.cc:260] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.087047 12785 ts_tablet_manager.cc:1434] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251209 20:55:24.089176 12789 tablet_bootstrap.cc:654] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.090015 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.090545 12399 raft_consensus.cc:2468] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.092155 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.092412 12398 raft_consensus.cc:3060] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.095067 12398 raft_consensus.cc:2468] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.095693 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.095955 12398 raft_consensus.cc:2393] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 157c709733f1484799756549aa456c5f in current term 1: Already voted for candidate 3cb2248952314571831b6958a0f0a7b4 in this term.
I20251209 20:55:24.097012 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a2925610d5e4948a0c2d93dd5072a2d" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.097224 12398 raft_consensus.cc:2393] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 157c709733f1484799756549aa456c5f in current term 1: Already voted for candidate 3cb2248952314571831b6958a0f0a7b4 in this term.
I20251209 20:55:24.098079 12470 leader_election.cc:304] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters: 157c709733f1484799756549aa456c5f
I20251209 20:55:24.098508 12800 raft_consensus.cc:2804] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.105667 12788 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.105888 12788 ts_tablet_manager.cc:1403] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.020s
I20251209 20:55:24.106624 12788 raft_consensus.cc:359] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.106817 12788 raft_consensus.cc:385] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.106922 12788 raft_consensus.cc:740] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.107160 12788 consensus_queue.cc:260] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
W20251209 20:55:24.108309 12606 leader_election.cc:343] T 4db0bbab30c24ce5befbdc1653251f3a P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: 4db0bbab30c24ce5befbdc1653251f3a
I20251209 20:55:24.109318 12606 leader_election.cc:304] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 157c709733f1484799756549aa456c5f; no voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:55:24.109638 12787 raft_consensus.cc:2749] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251209 20:55:24.110270 12802 raft_consensus.cc:493] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.110440 12802 raft_consensus.cc:515] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.111173 12802 leader_election.cc:290] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.111651 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b95f1e5c53a2417ca7ce58bfbafc2dcc" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.111709 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b95f1e5c53a2417ca7ce58bfbafc2dcc" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.111847 12672 raft_consensus.cc:2468] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:24.111871 12537 raft_consensus.cc:2468] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:24.112258 12332 leader_election.cc:304] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.112634 12802 raft_consensus.cc:2804] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.112743 12802 raft_consensus.cc:493] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.112859 12802 raft_consensus.cc:3060] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.114127 12788 ts_tablet_manager.cc:1434] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.008s user 0.001s sys 0.007s
I20251209 20:55:24.115814 12802 raft_consensus.cc:515] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.116189 12788 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:24.116693 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b95f1e5c53a2417ca7ce58bfbafc2dcc" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.116858 12537 raft_consensus.cc:3060] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
W20251209 20:55:24.117870 12052 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.132.252:42679, user_credentials={real_user=slave}} blocked reactor thread for 50702us
I20251209 20:55:24.118146 12800 raft_consensus.cc:697] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:24.118599 12800 consensus_queue.cc:237] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.119344 12537 raft_consensus.cc:2468] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:24.119706 12332 leader_election.cc:304] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.120023 12824 raft_consensus.cc:2804] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.124857 12802 leader_election.cc:290] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Requested vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.125274 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b95f1e5c53a2417ca7ce58bfbafc2dcc" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.125483 12672 raft_consensus.cc:3060] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.125537 12788 tablet_bootstrap.cc:654] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.128530 12672 raft_consensus.cc:2468] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:24.129360 12824 raft_consensus.cc:697] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Becoming Leader. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Running, Role: LEADER
I20251209 20:55:24.129704 12824 consensus_queue.cc:237] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.131517 12787 raft_consensus.cc:493] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.131695 12787 raft_consensus.cc:515] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.132251 12787 leader_election.cc:290] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.132939 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f187335da3a4b39b4c453eb470e99e8" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.133136 12537 raft_consensus.cc:2468] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.133482 12605 leader_election.cc:304] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.133736 12787 raft_consensus.cc:2804] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.133841 12787 raft_consensus.cc:493] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.133926 12787 raft_consensus.cc:3060] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.134696 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f187335da3a4b39b4c453eb470e99e8" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.134965 12398 raft_consensus.cc:2468] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.136466 12838 raft_consensus.cc:493] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.136620 12838 raft_consensus.cc:515] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.136811 12787 raft_consensus.cc:515] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.137410 12838 leader_election.cc:290] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.137712 12787 leader_election.cc:290] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.137817 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f187335da3a4b39b4c453eb470e99e8" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.138022 12536 raft_consensus.cc:3060] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.142426 12536 raft_consensus.cc:2468] T 0f187335da3a4b39b4c453eb470e99e8 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.142831 12605 leader_election.cc:304] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.143532 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e200643b18b43dd951a9af667bf1dca" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.143965 12672 raft_consensus.cc:2468] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:24.144488 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e200643b18b43dd951a9af667bf1dca" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.144773 12537 raft_consensus.cc:2468] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 0.
I20251209 20:55:24.145220 12332 leader_election.cc:304] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.145488 12787 raft_consensus.cc:2804] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.145498 12838 raft_consensus.cc:2804] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.145607 12838 raft_consensus.cc:493] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.145707 12838 raft_consensus.cc:3060] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.146034 12064 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Sending DeleteTablet for 3 replicas of tablet 8a2925610d5e4948a0c2d93dd5072a2d
I20251209 20:55:24.146224 12788 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.146445 12788 ts_tablet_manager.cc:1403] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20251209 20:55:24.146973 12064 catalog_manager.cc:6724] ReplaceTablet: tablet 8a2925610d5e4948a0c2d93dd5072a2d deleted and replaced by tablet 2f24e3afd3e24619a10ba98c02ae0899
I20251209 20:55:24.147027 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2f24e3afd3e24619a10ba98c02ae0899) from {username='slave'} at 127.0.0.1:54366
I20251209 20:55:24.147178 12788 raft_consensus.cc:359] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.147272 12791 tablet_replica.cc:333] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:24.147359 12788 raft_consensus.cc:385] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.147426 12788 raft_consensus.cc:740] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.147444 12791 raft_consensus.cc:2243] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.147608 12791 raft_consensus.cc:2272] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.147655 12788 consensus_queue.cc:260] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.148183 12788 ts_tablet_manager.cc:1434] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251209 20:55:24.148254 12838 raft_consensus.cc:515] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.148427 12791 ts_tablet_manager.cc:1916] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.148818 12838 leader_election.cc:290] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Requested vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.150177 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e200643b18b43dd951a9af667bf1dca" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.150350 12537 raft_consensus.cc:3060] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.151849 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f187335da3a4b39b4c453eb470e99e8" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.152040 12398 raft_consensus.cc:3060] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.152766 12537 raft_consensus.cc:2468] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:24.153120 12332 leader_election.cc:304] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.153388 12838 raft_consensus.cc:2804] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.153522 12838 raft_consensus.cc:697] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Becoming Leader. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Running, Role: LEADER
I20251209 20:55:24.153800 12838 consensus_queue.cc:237] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.154757 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2f24e3afd3e24619a10ba98c02ae0899) from {username='slave'} at 127.0.0.1:57366
I20251209 20:55:24.155018 12792 tablet_replica.cc:333] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:24.155128 12398 raft_consensus.cc:2468] T 0f187335da3a4b39b4c453eb470e99e8 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.155411 12791 ts_tablet_manager.cc:1929] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.155583 12791 log.cc:1199] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/8a2925610d5e4948a0c2d93dd5072a2d
I20251209 20:55:24.155995 12791 ts_tablet_manager.cc:1950] T 8a2925610d5e4948a0c2d93dd5072a2d P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:24.155512 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e200643b18b43dd951a9af667bf1dca" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.156847 12672 raft_consensus.cc:3060] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.156952 12054 catalog_manager.cc:4985] TS 157c709733f1484799756549aa456c5f (127.11.132.195:38447): tablet 8a2925610d5e4948a0c2d93dd5072a2d (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]) successfully deleted
I20251209 20:55:24.159938 12672 raft_consensus.cc:2468] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412a1acaab96447b97a475fb3f94fa12 in term 1.
I20251209 20:55:24.160786 12792 raft_consensus.cc:2243] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:24.161720 12792 pending_rounds.cc:70] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Trying to abort 1 pending ops.
I20251209 20:55:24.161880 12792 pending_rounds.cc:77] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7230725013994885120 op_type: NO_OP noop_request { }
I20251209 20:55:24.162052 12792 raft_consensus.cc:2889] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251209 20:55:24.162189 12792 raft_consensus.cc:2272] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.167591 12787 raft_consensus.cc:697] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [term 1 LEADER]: Becoming Leader. State: Replica: 157c709733f1484799756549aa456c5f, State: Running, Role: LEADER
I20251209 20:55:24.168035 12787 consensus_queue.cc:237] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.171427 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2f24e3afd3e24619a10ba98c02ae0899) from {username='slave'} at 127.0.0.1:51470
I20251209 20:55:24.171463 12792 ts_tablet_manager.cc:1916] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.171728 12793 tablet_replica.cc:333] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:24.171883 12793 raft_consensus.cc:2243] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.172016 12793 raft_consensus.cc:2272] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.176220 12793 ts_tablet_manager.cc:1916] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.177779 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 2f24e3afd3e24619a10ba98c02ae0899 (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.178241 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f24e3afd3e24619a10ba98c02ae0899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.182413 12793 ts_tablet_manager.cc:1929] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.182523 12793 log.cc:1199] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/8a2925610d5e4948a0c2d93dd5072a2d
I20251209 20:55:24.182893 12793 ts_tablet_manager.cc:1950] T 8a2925610d5e4948a0c2d93dd5072a2d P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:24.184841 12652 tablet_service.cc:1505] Processing CreateTablet for tablet 2f24e3afd3e24619a10ba98c02ae0899 (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.185266 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f24e3afd3e24619a10ba98c02ae0899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.185355 12792 ts_tablet_manager.cc:1929] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:24.185464 12792 log.cc:1199] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/8a2925610d5e4948a0c2d93dd5072a2d
I20251209 20:55:24.185770 12792 ts_tablet_manager.cc:1950] T 8a2925610d5e4948a0c2d93dd5072a2d P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:24.186666 12051 catalog_manager.cc:4985] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): tablet 8a2925610d5e4948a0c2d93dd5072a2d (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]) successfully deleted
I20251209 20:55:24.189263 12788 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:24.191431 12788 tablet_bootstrap.cc:654] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.191453 12785 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:24.193042 12785 tablet_bootstrap.cc:654] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.193936 12788 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.194068 12788 ts_tablet_manager.cc:1403] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251209 20:55:24.194887 12788 raft_consensus.cc:359] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.195068 12788 raft_consensus.cc:385] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.195173 12788 raft_consensus.cc:740] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.195398 12788 consensus_queue.cc:260] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.195919 12788 ts_tablet_manager.cc:1434] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251209 20:55:24.205011 12785 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:24.205142 12785 ts_tablet_manager.cc:1403] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20251209 20:55:24.205785 12785 raft_consensus.cc:359] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.205953 12785 raft_consensus.cc:385] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.206041 12785 raft_consensus.cc:740] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.206264 12785 consensus_queue.cc:260] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.206723 12785 ts_tablet_manager.cc:1434] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251209 20:55:24.212268 12800 raft_consensus.cc:493] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.212446 12800 raft_consensus.cc:515] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.203497 12064 catalog_manager.cc:5654] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193). New cstate: current_term: 1 leader_uuid: "412a1acaab96447b97a475fb3f94fa12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:24.213496 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.213697 12672 raft_consensus.cc:2468] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.213582 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.214099 12472 leader_election.cc:304] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.214398 12837 raft_consensus.cc:2804] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.214497 12837 raft_consensus.cc:493] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.214599 12837 raft_consensus.cc:3060] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
W20251209 20:55:24.214529 12470 leader_election.cc:343] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Illegal state: must be running to vote when last-logged opid is not known
I20251209 20:55:24.217741 12837 raft_consensus.cc:515] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.218755 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 2f24e3afd3e24619a10ba98c02ae0899 (DEFAULT_TABLE table=default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.218832 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.219009 12672 raft_consensus.cc:3060] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.219357 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f24e3afd3e24619a10ba98c02ae0899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.221674 12672 raft_consensus.cc:2468] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.222196 12472 leader_election.cc:304] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.222436 12837 leader_election.cc:290] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.222579 12837 raft_consensus.cc:2804] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.222705 12837 raft_consensus.cc:697] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:24.222992 12837 consensus_queue.cc:237] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.223783 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
W20251209 20:55:24.224511 12470 leader_election.cc:343] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Illegal state: must be running to vote when last-logged opid is not known
I20251209 20:55:24.226032 12800 leader_election.cc:290] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.230749 12052 catalog_manager.cc:4985] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): tablet 8a2925610d5e4948a0c2d93dd5072a2d (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]) successfully deleted
I20251209 20:55:24.241596 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a2925610d5e4948a0c2d93dd5072a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2f24e3afd3e24619a10ba98c02ae0899) from {username='slave'} at 127.0.0.1:57366
W20251209 20:55:24.242318 12051 catalog_manager.cc:4952] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): delete failed for tablet 8a2925610d5e4948a0c2d93dd5072a2d because the tablet was not found. No further retry: Not found: Tablet not found: 8a2925610d5e4948a0c2d93dd5072a2d
I20251209 20:55:24.242791 12064 catalog_manager.cc:5654] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 157c709733f1484799756549aa456c5f (127.11.132.195). New cstate: current_term: 1 leader_uuid: "157c709733f1484799756549aa456c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: HEALTHY } } }
I20251209 20:55:24.265152 12064 catalog_manager.cc:5654] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193). New cstate: current_term: 1 leader_uuid: "412a1acaab96447b97a475fb3f94fa12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:24.266130 12064 catalog_manager.cc:5654] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193). New cstate: current_term: 1 leader_uuid: "412a1acaab96447b97a475fb3f94fa12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:24.270841 12789 tablet_bootstrap.cc:492] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.271039 12789 ts_tablet_manager.cc:1403] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.188s user 0.002s sys 0.050s
I20251209 20:55:24.271890 12789 raft_consensus.cc:359] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.272085 12789 raft_consensus.cc:385] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.272154 12789 raft_consensus.cc:740] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.272383 12789 consensus_queue.cc:260] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.277339 12789 ts_tablet_manager.cc:1434] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251209 20:55:24.278551 12789 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:24.279734 12064 catalog_manager.cc:5654] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194). New cstate: current_term: 1 leader_uuid: "3cb2248952314571831b6958a0f0a7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:24.280395 12789 tablet_bootstrap.cc:654] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.292822 12064 master_service.cc:954] ReplaceTablet: received request to replace tablet b95f1e5c53a2417ca7ce58bfbafc2dcc from {username='slave'} at 127.0.0.1:53374
I20251209 20:55:24.300243 12064 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Sending DeleteTablet for 3 replicas of tablet b95f1e5c53a2417ca7ce58bfbafc2dcc
I20251209 20:55:24.301023 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc with delete_type TABLET_DATA_DELETED (replaced by tablet cb8c74dcd4374a0fa71a259d5961f7c4) from {username='slave'} at 127.0.0.1:51470
I20251209 20:55:24.301330 12793 tablet_replica.cc:333] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:24.301537 12793 raft_consensus.cc:2243] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:24.301901 12793 pending_rounds.cc:70] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Trying to abort 1 pending ops.
I20251209 20:55:24.302110 12793 pending_rounds.cc:77] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7230725014038118400 op_type: NO_OP noop_request { }
I20251209 20:55:24.302338 12793 raft_consensus.cc:2889] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251209 20:55:24.302488 12793 raft_consensus.cc:2272] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.303891 12793 ts_tablet_manager.cc:1916] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.307121 12837 raft_consensus.cc:493] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.307684 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc with delete_type TABLET_DATA_DELETED (replaced by tablet cb8c74dcd4374a0fa71a259d5961f7c4) from {username='slave'} at 127.0.0.1:54366
I20251209 20:55:24.308002 12791 tablet_replica.cc:333] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:24.308204 12791 raft_consensus.cc:2243] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.308286 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet b95f1e5c53a2417ca7ce58bfbafc2dcc with delete_type TABLET_DATA_DELETED (replaced by tablet cb8c74dcd4374a0fa71a259d5961f7c4) from {username='slave'} at 127.0.0.1:57366
I20251209 20:55:24.308357 12791 raft_consensus.cc:2272] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.308779 12792 tablet_replica.cc:333] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:24.308920 12791 ts_tablet_manager.cc:1916] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.308969 12792 raft_consensus.cc:2243] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:24.309325 12792 raft_consensus.cc:2272] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:24.307328 12837 raft_consensus.cc:515] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.310261 12837 leader_election.cc:290] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.310889 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.311092 12672 raft_consensus.cc:2468] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.311614 12472 leader_election.cc:304] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.311832 12793 ts_tablet_manager.cc:1929] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:24.311957 12793 log.cc:1199] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/b95f1e5c53a2417ca7ce58bfbafc2dcc
I20251209 20:55:24.312033 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.312377 12793 ts_tablet_manager.cc:1950] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
W20251209 20:55:24.312597 12470 leader_election.cc:343] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Illegal state: must be running to vote when last-logged opid is not known
I20251209 20:55:24.313306 12792 ts_tablet_manager.cc:1916] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:24.313455 12052 catalog_manager.cc:4985] TS 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]) successfully deleted
I20251209 20:55:24.316614 12791 ts_tablet_manager.cc:1929] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.316814 12791 log.cc:1199] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/b95f1e5c53a2417ca7ce58bfbafc2dcc
I20251209 20:55:24.317227 12791 ts_tablet_manager.cc:1950] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:24.317812 12837 raft_consensus.cc:2804] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.317961 12837 raft_consensus.cc:493] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.318090 12837 raft_consensus.cc:3060] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.318267 12054 catalog_manager.cc:4985] TS 157c709733f1484799756549aa456c5f (127.11.132.195:38447): tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]) successfully deleted
I20251209 20:55:24.320815 12837 raft_consensus.cc:515] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.321765 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
W20251209 20:55:24.322252 12470 leader_election.cc:343] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Illegal state: must be running to vote when last-logged opid is not known
I20251209 20:55:24.323112 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.323330 12672 raft_consensus.cc:3060] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.326673 12672 raft_consensus.cc:2468] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.329597 12652 tablet_service.cc:1505] Processing CreateTablet for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.329991 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8c74dcd4374a0fa71a259d5961f7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.332624 12517 tablet_service.cc:1505] Processing CreateTablet for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.333060 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8c74dcd4374a0fa71a259d5961f7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.336063 12472 leader_election.cc:304] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters: 412a1acaab96447b97a475fb3f94fa12
I20251209 20:55:24.336391 12800 raft_consensus.cc:2804] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.336512 12800 raft_consensus.cc:697] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:24.336757 12800 consensus_queue.cc:237] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.336874 12378 tablet_service.cc:1505] Processing CreateTablet for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (DEFAULT_TABLE table=default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:24.337360 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8c74dcd4374a0fa71a259d5961f7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:24.348263 12785 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:24.348711 12837 leader_election.cc:290] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.350636 12785 tablet_bootstrap.cc:654] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.354740 12064 catalog_manager.cc:6724] ReplaceTablet: tablet b95f1e5c53a2417ca7ce58bfbafc2dcc deleted and replaced by tablet cb8c74dcd4374a0fa71a259d5961f7c4
I20251209 20:55:24.356175 12064 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53374:
name: "default.Table_3d73b33061d7464da6fc7c9367ad163b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:24.361654 12785 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:24.361944 12785 ts_tablet_manager.cc:1403] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20251209 20:55:24.358587 12792 ts_tablet_manager.cc:1929] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251209 20:55:24.362989 12792 log.cc:1199] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/b95f1e5c53a2417ca7ce58bfbafc2dcc
I20251209 20:55:24.363456 12792 ts_tablet_manager.cc:1950] T b95f1e5c53a2417ca7ce58bfbafc2dcc P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:24.363343 12785 raft_consensus.cc:359] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.363750 12785 raft_consensus.cc:385] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.363883 12785 raft_consensus.cc:740] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.364352 12785 consensus_queue.cc:260] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.364473 12051 catalog_manager.cc:4985] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): tablet b95f1e5c53a2417ca7ce58bfbafc2dcc (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]) successfully deleted
I20251209 20:55:24.364969 12788 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:24.366703 12788 tablet_bootstrap.cc:654] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.367434 12785 ts_tablet_manager.cc:1434] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251209 20:55:24.373584 12789 tablet_bootstrap.cc:492] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.373747 12789 ts_tablet_manager.cc:1403] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.095s user 0.016s sys 0.030s
I20251209 20:55:24.374583 12789 raft_consensus.cc:359] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.374758 12789 raft_consensus.cc:385] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.374836 12789 raft_consensus.cc:740] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.375056 12789 consensus_queue.cc:260] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.379012 12789 ts_tablet_manager.cc:1434] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251209 20:55:24.380159 12787 raft_consensus.cc:493] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.380330 12787 raft_consensus.cc:515] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.380872 12787 leader_election.cc:290] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.381508 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c5cdf7ec11442e99363cb062365d38" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.381691 12398 raft_consensus.cc:2468] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.382208 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c5cdf7ec11442e99363cb062365d38" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.382330 12606 leader_election.cc:304] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.382421 12537 raft_consensus.cc:2468] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.382619 12787 raft_consensus.cc:2804] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.382720 12787 raft_consensus.cc:493] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.382843 12787 raft_consensus.cc:3060] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.383896 12788 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:24.384065 12788 ts_tablet_manager.cc:1403] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20251209 20:55:24.384814 12788 raft_consensus.cc:359] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.385020 12788 raft_consensus.cc:385] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.385087 12788 raft_consensus.cc:740] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.385470 12787 raft_consensus.cc:515] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.385341 12788 consensus_queue.cc:260] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.385656 12789 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:24.386406 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c5cdf7ec11442e99363cb062365d38" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.386575 12398 raft_consensus.cc:3060] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.387516 12789 tablet_bootstrap.cc:654] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.387830 12788 ts_tablet_manager.cc:1434] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251209 20:55:24.389499 12398 raft_consensus.cc:2468] T 19c5cdf7ec11442e99363cb062365d38 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.389927 12606 leader_election.cc:304] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.390285 12842 raft_consensus.cc:2804] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.390784 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "19c5cdf7ec11442e99363cb062365d38" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.390967 12537 raft_consensus.cc:3060] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.393368 12537 raft_consensus.cc:2468] T 19c5cdf7ec11442e99363cb062365d38 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.393803 12787 leader_election.cc:290] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.409976 12842 raft_consensus.cc:697] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [term 1 LEADER]: Becoming Leader. State: Replica: 157c709733f1484799756549aa456c5f, State: Running, Role: LEADER
I20251209 20:55:24.410362 12842 consensus_queue.cc:237] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.419349 12789 tablet_bootstrap.cc:492] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.420423 12789 ts_tablet_manager.cc:1403] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20251209 20:55:24.421242 12789 raft_consensus.cc:359] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.421444 12789 raft_consensus.cc:385] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.421527 12789 raft_consensus.cc:740] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.421814 12789 consensus_queue.cc:260] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.424752 12789 ts_tablet_manager.cc:1434] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251209 20:55:24.427515 12789 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:24.429031 12789 tablet_bootstrap.cc:654] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.436606 12789 tablet_bootstrap.cc:492] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.436736 12789 ts_tablet_manager.cc:1403] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20251209 20:55:24.437605 12789 raft_consensus.cc:359] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.437794 12789 raft_consensus.cc:385] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.437909 12789 raft_consensus.cc:740] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.438122 12789 consensus_queue.cc:260] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.441373 12789 ts_tablet_manager.cc:1434] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251209 20:55:24.443920 12789 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:24.445375 12789 tablet_bootstrap.cc:654] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:24.448424 12789 tablet_bootstrap.cc:492] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:24.448619 12789 ts_tablet_manager.cc:1403] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20251209 20:55:24.449208 12789 raft_consensus.cc:359] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.449412 12789 raft_consensus.cc:385] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:24.449505 12789 raft_consensus.cc:740] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.449780 12789 consensus_queue.cc:260] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.451809 12789 ts_tablet_manager.cc:1434] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
20:55:24.502 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_79b3eb661bba41bb99071683748470b5 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251209 20:55:24.505422 12063 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=b647dc20007a4c6cb80ac3d3432104d6]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_79b3eb661bba41bb99071683748470b5 already exists)
I20251209 20:55:24.505868 12063 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:53374 (request call id 6) took 1236 ms. Trace:
I20251209 20:55:24.505985 12063 rpcz_store.cc:276] 1209 20:55:23.268990 (+ 0us) service_pool.cc:168] Inserting onto call queue
1209 20:55:23.269081 (+ 91us) service_pool.cc:225] Handling call
1209 20:55:23.289624 (+ 20543us) catalog_manager.cc:2101] Acquired catalog manager lock
1209 20:55:23.290036 (+ 412us) catalog_manager.cc:2147] Created new table and tablet info
1209 20:55:24.505833 (+1215797us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":712580,"Hive Metastore.run_cpu_time_us":980,"Hive Metastore.run_wall_time_us":502294}
I20251209 20:55:24.507822 12063 catalog_manager.cc:5654] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194). New cstate: current_term: 1 leader_uuid: "3cb2248952314571831b6958a0f0a7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
F20251209 20:55:24.514884 12271 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7669278dcd google::LogMessage::Fail() at ??:0
@ 0x7f766927cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f76692787cc google::LogMessage::Flush() at ??:0
@ 0x7f7669278a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f767fc3faa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251209 20:55:24.531688 12063 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:24.531936 12063 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53374:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
@ 0x7f767fc3e351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f767fc45fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f767fc45e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f766bfed3f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f766f5f86db start_thread at ??:0
@ 0x7f766812971f clone at ??:0
W20251209 20:55:24.580423 12823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:24.580778 12823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:24.580878 12823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:24.592975 12823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:24.593052 12823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:24.593115 12823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:24.593163 12823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:24.608152 12823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=42787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:24.610101 12823 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:24.612186 12823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:24.633466 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:24.635509 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:24.636688 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:24.637430 12823 server_base.cc:1047] running on GCE node
I20251209 20:55:24.638266 12823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:24.640033 12823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:24.641358 12823 hybrid_clock.cc:648] HybridClock initialized: now 1765313724641119 us; error 222 us; skew 500 ppm
I20251209 20:55:24.641744 12823 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:24.644372 12823 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:24.645112 12823 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:24.645195 12823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:24.650287 12823 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20251209 20:55:24.653067 12882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:24.654388 12823 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:24.654536 12823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:24.655160 12823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:24.677719 12823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:24.678504 12823 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:24.678785 12823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:24.698776 12933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:24.698774 12823 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:24.700716 12823 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:24.704893 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 12823
I20251209 20:55:24.708341 12934 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:24.714152 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:24.715750 12800 raft_consensus.cc:493] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.716012 12800 raft_consensus.cc:515] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.716849 12800 leader_election.cc:290] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.717346 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9dfdd1ef1e5a4a60a5c9fcdf9e85a845" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.717571 12398 raft_consensus.cc:2468] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.717600 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9dfdd1ef1e5a4a60a5c9fcdf9e85a845" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.717851 12672 raft_consensus.cc:2468] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.718081 12470 leader_election.cc:304] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.718401 12800 raft_consensus.cc:2804] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.718505 12800 raft_consensus.cc:493] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.718617 12800 raft_consensus.cc:3060] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.721838 12800 raft_consensus.cc:515] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.723031 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9dfdd1ef1e5a4a60a5c9fcdf9e85a845" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.723263 12398 raft_consensus.cc:3060] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.725956 12398 raft_consensus.cc:2468] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.726369 12470 leader_election.cc:304] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:24.726687 12837 raft_consensus.cc:2804] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.726760 12800 leader_election.cc:290] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.726853 12837 raft_consensus.cc:697] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:24.727031 12934 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:24.727149 12837 consensus_queue.cc:237] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.727231 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9dfdd1ef1e5a4a60a5c9fcdf9e85a845" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.727424 12672 raft_consensus.cc:3060] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.731137 12672 raft_consensus.cc:2468] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.742415 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:24.743031 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap complete.
I20251209 20:55:24.744135 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:24.744325 12582 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:24.749720 12934 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:24.750495 12934 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:24.751183 12934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:24.752041 12940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:24.752333 12940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:24.753278 12934 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:24.761852 12824 raft_consensus.cc:493] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.762131 12824 raft_consensus.cc:515] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.762818 12824 leader_election.cc:290] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.763458 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.763808 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f24e3afd3e24619a10ba98c02ae0899" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.764096 12672 raft_consensus.cc:2393] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412a1acaab96447b97a475fb3f94fa12 in current term 1: Already voted for candidate 3cb2248952314571831b6958a0f0a7b4 in this term.
I20251209 20:55:24.764807 12335 leader_election.cc:304] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412a1acaab96447b97a475fb3f94fa12; no voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:24.765450 12824 raft_consensus.cc:3060] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.768365 12824 raft_consensus.cc:2749] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251209 20:55:24.791991 12955 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:24.792160 12955 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: success
I20251209 20:55:24.797502 12955 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: acquiring CA information for follower catalog manager: success
I20251209 20:55:24.801144 12955 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:24.802034 12837 raft_consensus.cc:493] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.802271 12837 raft_consensus.cc:515] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.803120 12837 leader_election.cc:290] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.803689 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73f0546be0c64a6287404fd0f3a9d427" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.803689 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73f0546be0c64a6287404fd0f3a9d427" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:24.803913 12398 raft_consensus.cc:2468] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.803913 12672 raft_consensus.cc:2468] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:24.804394 12472 leader_election.cc:304] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.804737 12837 raft_consensus.cc:2804] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.804819 12837 raft_consensus.cc:493] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.804913 12837 raft_consensus.cc:3060] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.807488 12837 raft_consensus.cc:515] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.808599 12837 leader_election.cc:290] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:24.808682 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73f0546be0c64a6287404fd0f3a9d427" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:24.808655 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73f0546be0c64a6287404fd0f3a9d427" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.808842 12398 raft_consensus.cc:3060] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.808842 12672 raft_consensus.cc:3060] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.811511 12672 raft_consensus.cc:2468] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.811511 12398 raft_consensus.cc:2468] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:24.811966 12472 leader_election.cc:304] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.812252 12837 raft_consensus.cc:2804] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.812640 12582 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:24.812866 12842 raft_consensus.cc:493] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:24.813006 12842 raft_consensus.cc:515] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.813326 12837 raft_consensus.cc:697] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:24.813630 12842 leader_election.cc:290] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.814020 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb8c74dcd4374a0fa71a259d5961f7c4" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:24.814024 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb8c74dcd4374a0fa71a259d5961f7c4" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:24.814221 12537 raft_consensus.cc:2468] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.814240 12398 raft_consensus.cc:2468] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:24.814564 12605 leader_election.cc:304] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.814544 12837 consensus_queue.cc:237] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.814973 12842 raft_consensus.cc:2804] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:24.815061 12842 raft_consensus.cc:493] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:24.815177 12842 raft_consensus.cc:3060] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.817622 12842 raft_consensus.cc:515] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.819355 12537 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb8c74dcd4374a0fa71a259d5961f7c4" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:24.819530 12537 raft_consensus.cc:3060] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.822280 12537 raft_consensus.cc:2468] T cb8c74dcd4374a0fa71a259d5961f7c4 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.822695 12605 leader_election.cc:304] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:24.823005 12853 raft_consensus.cc:2804] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:24.818493 12398 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb8c74dcd4374a0fa71a259d5961f7c4" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:24.823165 12853 raft_consensus.cc:697] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [term 1 LEADER]: Becoming Leader. State: Replica: 157c709733f1484799756549aa456c5f, State: Running, Role: LEADER
I20251209 20:55:24.823377 12398 raft_consensus.cc:3060] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:24.823501 12853 consensus_queue.cc:237] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:24.823163 12842 leader_election.cc:290] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.830137 12898 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:49961; Asking this server to re-register.
I20251209 20:55:24.831043 12582 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:24.831746 12398 raft_consensus.cc:2468] T cb8c74dcd4374a0fa71a259d5961f7c4 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:24.833763 12899 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:24.834682 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:24.834896 12717 heartbeater.cc:499] Master 127.11.132.252:42679 was elected leader, sending a full tablet report...
I20251209 20:55:24.836673 12899 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:44527; Asking this server to re-register.
I20251209 20:55:24.837602 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:24.838979 12899 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
W20251209 20:55:24.863194 12724 connection.cc:537] client connection to 127.11.132.254:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:24.863241 12469 connection.cc:537] client connection to 127.11.132.254:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:24.864125 12724 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.254:40601)
W20251209 20:55:24.864333 12724 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.254:40601)
W20251209 20:55:24.864995 12724 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.254:40601)
W20251209 20:55:24.865738 12774 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 20:55:24.864238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.132.254:40601 (local address 127.11.132.195:46685)
1209 20:55:24.864469 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1209 20:55:24.864486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 20:55:24.864587 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 20:55:24.864878 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
W20251209 20:55:24.865919 12724 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.254:40601)
W20251209 20:55:24.867379 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 20:55:24.776245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.132.254:40601 (local address 127.0.0.1:53384)
1209 20:55:24.784420 (+ 8175us) negotiation.cc:107] Waiting for socket to connect
1209 20:55:24.784490 (+ 70us) client_negotiation.cc:174] Beginning negotiation
1209 20:55:24.784763 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 20:55:24.866068 (+ 81305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7711,"thread_start_us":8581,"threads_started":1}
I20251209 20:55:24.869650 12823 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20251209 20:55:24.874720 12823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:24.960299 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:24.962512 12899 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:52235; Asking this server to re-register.
I20251209 20:55:24.963304 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:24.964696 12899 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:25.004890 12824 consensus_queue.cc:1048] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.017675 12965 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:25.017860 12965 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:25.021221 12965 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:25.023615 12853 consensus_queue.cc:1048] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.033432 12909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:25.034183 12909 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc9184e29d6430ca18a3fe7984c725c in term 1.
I20251209 20:55:25.034801 12122 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482; no voters:
I20251209 20:55:25.035100 12965 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251209 20:55:25.035230 12965 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Starting leader election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:25.035337 12965 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 2 FOLLOWER]: Advancing to term 3
I20251209 20:55:25.038292 12965 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:25.039286 12965 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:25.039543 12909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482"
I20251209 20:55:25.039783 12909 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 1 FOLLOWER]: Advancing to term 3
I20251209 20:55:25.043195 12824 consensus_queue.cc:1048] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.048571 12842 consensus_queue.cc:1048] T 0f187335da3a4b39b4c453eb470e99e8 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.058152 12909 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc9184e29d6430ca18a3fe7984c725c in term 3.
W20251209 20:55:25.062317 12121 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:25.063419 12122 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482; no voters:
I20251209 20:55:25.063818 12965 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Leader election won for term 3
I20251209 20:55:25.064278 12965 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 LEADER]: Becoming Leader. State: Replica: 1cc9184e29d6430ca18a3fe7984c725c, State: Running, Role: LEADER
I20251209 20:55:25.064836 12965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
W20251209 20:55:25.066804 12121 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:25.066788 12976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc9184e29d6430ca18a3fe7984c725c. Latest consensus state: current_term: 3 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:25.068323 12976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:25.071081 12977 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:25.073971 12977 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:25.074286 12824 consensus_queue.cc:1048] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.074460 12977 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:25.074874 12977 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:25.075373 12977 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:25.076946 12977 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:25.077519 12977 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:25.078008 12977 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:25.078552 12977 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:25.079107 12977 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:25.082542 12977 tablet_loader.cc:96] loaded metadata for tablet 0e200643b18b43dd951a9af667bf1dca (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:25.083216 12977 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:25.084606 12977 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:25.085048 12977 tablet_loader.cc:96] loaded metadata for tablet 2f24e3afd3e24619a10ba98c02ae0899 (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:25.085752 12977 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:25.086166 12977 tablet_loader.cc:96] loaded metadata for tablet 73f0546be0c64a6287404fd0f3a9d427 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:25.086898 12977 tablet_loader.cc:96] loaded metadata for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:25.087669 12977 tablet_loader.cc:96] loaded metadata for tablet c30b153a590a4fc488749b55b2e6ed29 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:25.088167 12977 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:25.088515 12977 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:25.090484 12977 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:25.090579 12977 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:25.092293 12977 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:25.092916 12824 consensus_queue.cc:1048] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.093449 12977 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Loaded TSK: 0
I20251209 20:55:25.095337 12977 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:25.096544 12977 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:25.098376 12977 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:25.107148 12446 heartbeater.cc:499] Master 127.11.132.253:34647 was elected leader, sending a full tablet report...
I20251209 20:55:25.108409 12584 heartbeater.cc:499] Master 127.11.132.253:34647 was elected leader, sending a full tablet report...
I20251209 20:55:25.113873 12137 catalog_manager.cc:5654] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194). New cstate: current_term: 1 leader_uuid: "3cb2248952314571831b6958a0f0a7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:25.114724 12137 catalog_manager.cc:5654] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194). New cstate: current_term: 1 leader_uuid: "3cb2248952314571831b6958a0f0a7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20251209 20:55:25.119930 12134 master_service.cc:954] ReplaceTablet: received request to replace tablet 73f0546be0c64a6287404fd0f3a9d427 from {username='slave'} at 127.0.0.1:60296
I20251209 20:55:25.120751 12132 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60296:
name: "default.Table_cf01c2d93fcb46cba2bbea670526b180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:25.121078 12133 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e200643b18b43dd951a9af667bf1dca from {username='slave'} at 127.0.0.1:60296
I20251209 20:55:25.120671 12135 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60296:
name: "default.Table_671e465611be47418f159b3c5ee7aa16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:25.122174 12131 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60296:
name: "default.Table_b7b340bc7e5b46b39332ec8149739649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:25.122817 12129 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60296:
name: "default.Table_afc836c7c2664ec0bd5f96ce0208fa81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:25.123119 12128 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60296:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:25.123366 12128 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60296:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:25.122129 12130 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60296:
name: "default.Table_dcb97bd9ec28454eb4f5627100a4e5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251209 20:55:25.137051 12312 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f6449e35dcd google::LogMessage::Fail() at ??:0
@ 0x7f6449e39b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6449e357cc google::LogMessage::Flush() at ??:0
I20251209 20:55:25.143977 12136 master_service.cc:954] ReplaceTablet: received request to replace tablet c30b153a590a4fc488749b55b2e6ed29 from {username='slave'} at 127.0.0.1:60296
@ 0x7f6449e35a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f64607fcaa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251209 20:55:25.161353 12909 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Refusing update from remote peer 1cc9184e29d6430ca18a3fe7984c725c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20251209 20:55:25.162240 12965 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f64607fb351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
W20251209 20:55:25.164455 12121 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c -> Peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Couldn't send request to peer ec484016839448df924d26e6175a3176. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f6460802fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f6460802e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f644cbaa3f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f64501b56db start_thread at ??:0
@ 0x7f6448ce671f clone at ??:0
I20251209 20:55:25.199004 12940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc9184e29d6430ca18a3fe7984c725c. Latest consensus state: current_term: 3 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:25.199257 12940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:25.199656 12988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:25.199940 12988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:25.228713 12984 mvcc.cc:204] Tried to move back new op lower bound from 7230725015591821312 to 7230725010247274496. Current Snapshot: MvccSnapshot[applied={T|T < 7230725015591821312 or (T in {7230725015591821312})}]
I20251209 20:55:25.238191 12842 consensus_queue.cc:1048] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.243510 12719 heartbeater.cc:499] Master 127.11.132.253:34647 was elected leader, sending a full tablet report...
I20251209 20:55:25.246038 12853 consensus_queue.cc:1048] T 19c5cdf7ec11442e99363cb062365d38 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.258039 12824 raft_consensus.cc:493] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:25.258194 12824 raft_consensus.cc:515] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:25.258764 12824 leader_election.cc:290] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:25.259214 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:25.259338 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "73119767050448079a74b823ea6587f5" candidate_uuid: "412a1acaab96447b97a475fb3f94fa12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:25.259608 12672 raft_consensus.cc:2393] T 73119767050448079a74b823ea6587f5 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 412a1acaab96447b97a475fb3f94fa12 in current term 1: Already voted for candidate 3cb2248952314571831b6958a0f0a7b4 in this term.
I20251209 20:55:25.260051 12335 leader_election.cc:304] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412a1acaab96447b97a475fb3f94fa12; no voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:25.260421 12824 raft_consensus.cc:3060] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:25.262332 12824 raft_consensus.cc:2749] T 73119767050448079a74b823ea6587f5 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251209 20:55:25.285307 12837 consensus_queue.cc:1048] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.301558 12836 consensus_queue.cc:1048] T 73119767050448079a74b823ea6587f5 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251209 20:55:25.411281 12606 connection.cc:537] client connection to 127.11.132.253:34647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:25.412762 12333 connection.cc:537] client connection to 127.11.132.253:34647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:25.412990 12883 connection.cc:537] server connection from 127.0.0.1:38920 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251209 20:55:25.447438 12837 consensus_queue.cc:1048] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.452390 12837 consensus_queue.cc:1048] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.543210 12837 consensus_queue.cc:1048] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.548525 13012 consensus_queue.cc:1048] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.582340 12842 consensus_queue.cc:1048] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.587026 12842 consensus_queue.cc:1048] T cb8c74dcd4374a0fa71a259d5961f7c4 P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.667086 12988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:25.667264 12988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:25.668898 12988 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:25.669958 12883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:25.671042 12884 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:25.671245 12883 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:25.671396 12883 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:25.671741 12988 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:25.775445 13012 consensus_queue.cc:1048] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:25.780241 12836 consensus_queue.cc:1048] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:26.240922 13024 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:26.241101 13024 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:26.241806 13024 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:26.243523 12883 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:26.243732 12884 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:26.243882 12884 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:26.244132 13024 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:27.036250 11795 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu pid 12823
I20251209 20:55:27.042203 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20251209 20:55:27.149072 11795 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu with pid 12823
I20251209 20:55:28.103221 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=42787
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:28.406656 13050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:28.406927 13050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:28.407040 13050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:28.415294 13050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:28.415372 13050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:28.415442 13050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:28.415481 13050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:28.424805 13050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=42787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:28.426195 13050 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:28.427841 13050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:28.435175 13063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:28.435194 13064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:28.435676 13066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:28.436112 13050 server_base.cc:1047] running on GCE node
I20251209 20:55:28.436704 13050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:28.437776 13050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:28.438970 13050 hybrid_clock.cc:648] HybridClock initialized: now 1765313728438921 us; error 38 us; skew 500 ppm
I20251209 20:55:28.439285 13050 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:28.441177 13050 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:28.441828 13050 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:28.441924 13050 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:28.447124 13050 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20251209 20:55:28.450132 13072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:28.451292 13050 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251209 20:55:28.451438 13050 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:28.452100 13050 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:28.476538 13050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:28.477218 13050 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:28.477437 13050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:28.496467 13050 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:28.496536 13123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:28.497876 13050 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:28.499679 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13050
I20251209 20:55:28.504846 13124 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:28.507647 13124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:28.517241 13124 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:28.542052 13124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251209 20:55:28.542619 13124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap complete.
I20251209 20:55:28.546991 13124 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:28.549223 13124 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:28.549798 13124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:28.550472 13129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:28.550685 13129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:28.551358 13124 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:28.573484 13143 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:28.573611 13143 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: success
I20251209 20:55:28.577250 13143 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: acquiring CA information for follower catalog manager: success
I20251209 20:55:28.579867 13143 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:28.576268 13050 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:55:28.581353 13050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:28.758471 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:28.761616 13089 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:49513; Asking this server to re-register.
I20251209 20:55:28.762291 12582 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:28.764076 13089 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:28.815438 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:28.816923 13089 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:35149; Asking this server to re-register.
I20251209 20:55:28.817436 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:28.818454 13089 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:28.918237 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:28.919519 13089 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:40435; Asking this server to re-register.
I20251209 20:55:28.920008 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:28.921175 13089 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:28.959196 13129 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:28.959424 13129 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:28.961122 13129 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:28.961601 13073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:28.962961 13073 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:28.963166 13074 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:28.963315 13074 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:28.963629 13129 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:29.699539 13158 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:29.699721 13158 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:29.700321 13158 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:29.702147 13074 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:29.702323 13073 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:29.702430 13073 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:29.702674 13158 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:30.642254 11795 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu pid 13050
I20251209 20:55:30.648239 13119 generic_service.cc:196] Checking for leaks (request via RPC)
I20251209 20:55:30.748782 13175 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:30.749011 13175 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:30.749078 11795 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu with pid 13050
I20251209 20:55:31.314088 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=42787
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:31.618881 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:31.619369 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:31.619495 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:31.627312 13182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:31.627379 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:31.627458 13182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:31.627529 13182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:31.636427 13182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=42787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:31.637809 13182 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:31.639629 13182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:31.647209 13189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:31.647171 13188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:31.647617 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:31.648566 13182 server_base.cc:1047] running on GCE node
I20251209 20:55:31.649317 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:31.650774 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:31.651974 13182 hybrid_clock.cc:648] HybridClock initialized: now 1765313731651921 us; error 45 us; skew 500 ppm
I20251209 20:55:31.652299 13182 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:31.654315 13182 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:31.655062 13182 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:31.655148 13182 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:31.660075 13182 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:31.662587 13197 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:31.663833 13182 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251209 20:55:31.663973 13182 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:31.664700 13182 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:31.680009 13182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:31.680716 13182 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:31.680968 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:31.699003 13182 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:31.699040 13248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:31.700470 13182 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:31.706833 13249 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:31.709877 13249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:31.710654 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13182
I20251209 20:55:31.721616 13249 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:31.756465 13249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251209 20:55:31.757095 13249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap complete.
I20251209 20:55:31.762523 13249 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:31.765013 13249 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:31.765615 13249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:31.766319 13255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:31.766554 13255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:31.767266 13249 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:31.784036 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:31.787847 13214 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:43999; Asking this server to re-register.
I20251209 20:55:31.788800 12582 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:31.791239 13214 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:31.798930 13269 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:31.799083 13269 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: success
I20251209 20:55:31.803362 13269 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: acquiring CA information for follower catalog manager: success
I20251209 20:55:31.807006 13269 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:31.803478 13182 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:55:31.809021 13182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:31.832211 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:31.833532 13214 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:38861; Asking this server to re-register.
I20251209 20:55:31.834096 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:31.835422 13214 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:31.935273 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:31.936497 13214 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:48001; Asking this server to re-register.
I20251209 20:55:31.937111 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:31.938436 13214 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:32.355345 13275 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:32.355624 13275 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:32.357337 13275 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:32.358366 13199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:32.359747 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:32.359943 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:32.360085 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:32.360416 13275 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:33.004962 13287 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:33.005152 13287 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:33.005878 13287 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:33.008636 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:33.008816 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:33.008942 13199 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:33.009179 13287 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:33.829838 13302 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:33.829998 13302 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:33.830564 13302 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:33.832509 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:33.832654 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:33.832800 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:33.833040 13302 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:34.605312 13314 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:34.605484 13314 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:34.606045 13314 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:34.608224 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:34.608372 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:34.608496 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:34.608755 13314 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251209 20:55:34.975270 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--webserver_interface=127.11.132.253
--webserver_port=34115
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:35.260201 13325 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:35.260411 13325 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:35.261161 13325 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:35.263391 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:35.263581 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:35.263733 13199 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:35.264025 13325 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251209 20:55:35.296597 13324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:35.296850 13324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:35.296962 13324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:35.304443 13324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:35.304529 13324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:35.304625 13324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:35.304699 13324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:35.314371 13324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.132.253
--webserver_port=34115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:35.315847 13324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:35.317462 13324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:35.324843 13336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:35.330717 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:35.331187 13338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:35.333577 13324 server_base.cc:1047] running on GCE node
I20251209 20:55:35.334232 13324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:35.335464 13324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:35.336642 13324 hybrid_clock.cc:648] HybridClock initialized: now 1765313735336603 us; error 54 us; skew 500 ppm
I20251209 20:55:35.336925 13324 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:35.338764 13324 webserver.cc:492] Webserver started at http://127.11.132.253:34115/ using document root <none> and password file <none>
I20251209 20:55:35.339438 13324 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:35.339553 13324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:35.344424 13324 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20251209 20:55:35.346850 13346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:35.348167 13324 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251209 20:55:35.348310 13324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
uuid: "1cc9184e29d6430ca18a3fe7984c725c"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:35.349030 13324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:35.360256 13324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:35.360873 13324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:35.361129 13324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:35.379379 13324 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.253:34647
I20251209 20:55:35.379415 13397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.253:34647 every 8 connection(s)
I20251209 20:55:35.380738 13324 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
I20251209 20:55:35.383703 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13324
I20251209 20:55:35.387996 13398 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:35.390972 13398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap starting.
I20251209 20:55:35.402007 13398 log.cc:826] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:35.429172 13398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251209 20:55:35.429904 13398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap complete.
I20251209 20:55:35.435710 12446 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:35.436422 13398 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:35.438812 13398 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9184e29d6430ca18a3fe7984c725c, State: Initialized, Role: FOLLOWER
I20251209 20:55:35.439348 13398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:35.440011 13405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:35.440240 13405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:35.440909 13398 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:35.455209 13419 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:35.455303 13419 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: loading cluster ID for follower catalog manager: success
I20251209 20:55:35.451581 13324 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:55:35.457304 13324 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:35.458702 13419 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: acquiring CA information for follower catalog manager: success
I20251209 20:55:35.461724 13419 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:35.775923 12584 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:35.776108 12584 heartbeater.cc:499] Master 127.11.132.253:34647 was elected leader, sending a full tablet report...
I20251209 20:55:35.778379 13363 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:36375; Asking this server to re-register.
I20251209 20:55:35.779063 12584 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:35.780798 13363 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:35.838249 12719 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:35.838419 12719 heartbeater.cc:499] Master 127.11.132.253:34647 was elected leader, sending a full tablet report...
I20251209 20:55:35.839928 13363 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:38405; Asking this server to re-register.
I20251209 20:55:35.840482 12719 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:35.841744 13363 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:35.940614 13363 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:51763; Asking this server to re-register.
I20251209 20:55:35.941182 12446 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:35.942211 13363 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:36.032024 13428 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:36.032284 13428 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:36.033954 13428 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:36.034859 13347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:36.036262 13347 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:36.040405 13224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:36.040812 13224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cc9184e29d6430ca18a3fe7984c725c in term 3.
I20251209 20:55:36.041433 13348 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482; no voters: ec484016839448df924d26e6175a3176
I20251209 20:55:36.041756 13428 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251209 20:55:36.041882 13428 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:36.042001 13428 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 3 FOLLOWER]: Advancing to term 4
I20251209 20:55:36.045230 13428 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:36.045755 13428 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:36.046283 13224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482"
I20251209 20:55:36.046484 13224 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 3 FOLLOWER]: Advancing to term 4
W20251209 20:55:36.046921 13347 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:36.049703 13224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cc9184e29d6430ca18a3fe7984c725c in term 4.
I20251209 20:55:36.050047 13348 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482; no voters: ec484016839448df924d26e6175a3176
I20251209 20:55:36.050319 13428 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Leader election won for term 4
I20251209 20:55:36.050659 13428 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 LEADER]: Becoming Leader. State: Replica: 1cc9184e29d6430ca18a3fe7984c725c, State: Running, Role: LEADER
I20251209 20:55:36.051049 13428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.38, Last appended by leader: 38, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:36.053507 13433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc9184e29d6430ca18a3fe7984c725c. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.053738 13433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: LEADER
W20251209 20:55:36.456944 13347 consensus_peers.cc:597] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c -> Peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Couldn't send request to peer ec484016839448df924d26e6175a3176. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:36.562449 13224 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Refusing update from remote peer 1cc9184e29d6430ca18a3fe7984c725c: Log matching property violated. Preceding OpId in replica: term: 3 index: 38. Preceding OpId from leader: term: 4 index: 39. (index mismatch)
I20251209 20:55:36.563373 13433 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 36, Time since last communication: 0.000s
I20251209 20:55:36.566069 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc9184e29d6430ca18a3fe7984c725c. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.566258 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:36.575815 13440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.576037 13435 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:36.576087 13440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:36.576037 13446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.576455 13446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:36.576637 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.576761 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:36.578327 13435 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:36.578768 13435 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:36.579124 13435 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:36.579422 13435 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:36.579797 13435 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:36.579746 13433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1cc9184e29d6430ca18a3fe7984c725c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:36.579967 13433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:36.580245 13435 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:36.580552 13435 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:36.580858 13435 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:36.581220 13435 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:36.582913 13435 tablet_loader.cc:96] loaded metadata for tablet 0e200643b18b43dd951a9af667bf1dca (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:36.583271 13435 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:36.583608 13435 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:36.583914 13435 tablet_loader.cc:96] loaded metadata for tablet 2f24e3afd3e24619a10ba98c02ae0899 (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:36.584379 13435 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:36.584614 13435 tablet_loader.cc:96] loaded metadata for tablet 73f0546be0c64a6287404fd0f3a9d427 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:36.585053 13435 tablet_loader.cc:96] loaded metadata for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:36.585435 13435 tablet_loader.cc:96] loaded metadata for tablet c30b153a590a4fc488749b55b2e6ed29 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:36.585702 13435 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:36.585906 13435 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:36.586712 13435 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:36.586776 13435 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:36.587858 13435 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:36.588584 13435 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Loaded TSK: 0
I20251209 20:55:36.589308 13435 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:36.590112 13435 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:36.590819 13435 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:36.675734 13363 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59190:
name: "default.Table_1b5967177d41405ababac4e174a63fc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:36.675884 13362 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_e2d4d7aea5fd4d41a0b084f0b7726b94" } new_table_name: "default.Table_a55cc08bf00c49778a226eba00e323f2" modify_external_catalogs: true
I20251209 20:55:36.676415 13358 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59190:
name: "default.Table_3d73b33061d7464da6fc7c9367ad163b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:36.676627 13354 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59190:
name: "default.Table_cf01c2d93fcb46cba2bbea670526b180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:36.677151 13360 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:36.677361 13360 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:36.677333 13355 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59190:
name: "default.Table_b7b340bc7e5b46b39332ec8149739649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:36.677918 13357 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_75ebe9a697e04984b340737cc08cb914" } new_table_name: "default.Table_8f50250b772546b68acb72dc72554936" modify_external_catalogs: true
I20251209 20:55:36.678189 13361 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_27e25321226d4880ad3b53e9936fd7a8" } new_table_name: "default.Table_855e1033632d4d57b523f0e18d938547" modify_external_catalogs: true
I20251209 20:55:36.678377 13359 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef" } new_table_name: "default.Table_5d01adbefb7248b1a99539733a5c4687" modify_external_catalogs: true
I20251209 20:55:36.676450 13356 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59190:
table { table_name: "default.Table_6f837998c1284f3c9172483c60da8813" } new_table_name: "default.Table_581f8285cf154582b0b9bb5096b8ae7a" modify_external_catalogs: true
F20251209 20:55:36.682613 13418 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f15df6efdcd google::LogMessage::Fail() at ??:0
@ 0x7f15df6f3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f15df6ef7cc google::LogMessage::Flush() at ??:0
@ 0x7f15df6efa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f15f60b6aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f15f60b5351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f15f60bcfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f15f60bce49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f15e24643f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f15e5a6f6db start_thread at ??:0
@ 0x7f15de5a071f clone at ??:0
I20251209 20:55:36.810693 13182 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:55:36.810935 13182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:55:36.866245 12606 connection.cc:537] client connection to 127.11.132.253:34647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:36.866288 12470 connection.cc:537] client connection to 127.11.132.253:34647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:36.866964 12725 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.253:34647)
W20251209 20:55:36.867249 12725 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.253:34647)
W20251209 20:55:36.867436 12725 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.253:34647)
W20251209 20:55:36.868105 12725 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.253:34647)
I20251209 20:55:37.068749 13438 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:37.068893 13438 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:37.069487 13438 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:37.071177 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:37.071352 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:37.071450 13199 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:37.071682 13438 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251209 20:55:37.821538 13467 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:37.821692 13467 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:37.822250 13467 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:37.822944 13198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20251209 20:55:37.824079 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:37.824291 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:37.824391 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:37.824571 13467 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251209 20:55:38.269132 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--webserver_interface=127.11.132.254
--webserver_port=39627
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:38.398171 13474 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:38.398355 13474 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:38.398970 13474 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:38.400971 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:38.401168 13198 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:38.401281 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482; no voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176
I20251209 20:55:38.401539 13474 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251209 20:55:38.587033 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:38.587425 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:38.587545 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:38.598552 13472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:38.598663 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:38.598732 13472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:38.598781 13472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:38.609882 13472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.132.254
--webserver_port=39627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:38.613071 13472 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:38.615125 13472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:38.622360 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:38.623215 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:38.624344 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:38.624948 13472 server_base.cc:1047] running on GCE node
I20251209 20:55:38.625622 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:38.626876 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:38.628072 13472 hybrid_clock.cc:648] HybridClock initialized: now 1765313738628030 us; error 35 us; skew 500 ppm
I20251209 20:55:38.628429 13472 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:38.630456 13472 webserver.cc:492] Webserver started at http://127.11.132.254:39627/ using document root <none> and password file <none>
I20251209 20:55:38.631134 13472 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:38.631255 13472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:38.636209 13472 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:38.638872 13491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:38.640136 13472 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:38.640296 13472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
uuid: "ec484016839448df924d26e6175a3176"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:38.640923 13472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:38.653259 13472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:38.653970 13472 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:38.654238 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:38.674125 13472 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.254:40601
I20251209 20:55:38.674144 13542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.254:40601 every 8 connection(s)
I20251209 20:55:38.675531 13472 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
I20251209 20:55:38.681758 13543 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:38.684767 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap starting.
I20251209 20:55:38.685586 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13472
I20251209 20:55:38.698582 13543 log.cc:826] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:38.729311 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:38.729878 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap complete.
I20251209 20:55:38.734891 13543 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:38.735509 13543 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec484016839448df924d26e6175a3176, State: Initialized, Role: FOLLOWER
I20251209 20:55:38.736181 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:38.737161 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:38.737521 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:38.738399 13543 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:38.758494 13561 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:38.758566 13561 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: loading cluster ID for follower catalog manager: success
I20251209 20:55:38.756314 13472 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:38.760746 13472 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:38.762737 13561 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: acquiring CA information for follower catalog manager: success
I20251209 20:55:38.765618 13561 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:38.783644 12583 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:38.787457 13508 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:46809; Asking this server to re-register.
I20251209 20:55:38.788267 12583 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:38.789865 13508 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:38.845414 12718 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:38.847797 13508 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:50909; Asking this server to re-register.
I20251209 20:55:38.848400 12718 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:38.849706 13508 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:38.944545 12445 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:38.946368 13508 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:46221; Asking this server to re-register.
I20251209 20:55:38.946942 12445 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:38.948459 13508 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:39.244366 13566 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:39.244621 13566 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:39.246421 13566 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:39.247110 13493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:39.248821 13493 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:39.252038 13224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:39.252279 13224 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec484016839448df924d26e6175a3176 for earlier term 3. Current term is 4.
I20251209 20:55:39.252912 13493 leader_election.cc:400] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec484016839448df924d26e6175a3176 for earlier term 3. Current term is 4.
I20251209 20:55:39.252990 13493 leader_election.cc:403] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251209 20:55:39.253266 13566 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 2 FOLLOWER]: Advancing to term 4
I20251209 20:55:39.256446 13566 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec484016839448df924d26e6175a3176 for earlier term 3. Current term is 4.
I20251209 20:55:39.861224 13574 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:39.861373 13574 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:39.861927 13574 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:39.862793 13224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 5 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "752a47ea6a344edba26b69df6ea3d482" is_pre_election: true
I20251209 20:55:39.863142 13224 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec484016839448df924d26e6175a3176 for term 5 because replica has last-logged OpId of term: 4 index: 39, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 36.
W20251209 20:55:39.864425 13493 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:39.864569 13493 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:39.865072 13574 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251209 20:55:39.988255 13580 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:39.988423 13580 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:39.989001 13580 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:39.991274 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:39.995642 13518 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:39.996052 13518 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 4.
I20251209 20:55:39.996605 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c
I20251209 20:55:39.996866 13580 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251209 20:55:39.996960 13580 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cc9184e29d6430ca18a3fe7984c725c)
I20251209 20:55:39.997051 13580 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 4 FOLLOWER]: Advancing to term 5
I20251209 20:55:39.998769 13580 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:39.999284 13580 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:39.999666 13518 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176"
I20251209 20:55:39.999842 13518 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 4 FOLLOWER]: Advancing to term 5
W20251209 20:55:40.000389 13199 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:40.001785 13518 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 5.
I20251209 20:55:40.002110 13198 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c
I20251209 20:55:40.002372 13580 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Leader election won for term 5
I20251209 20:55:40.002741 13580 raft_consensus.cc:697] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 LEADER]: Becoming Leader. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Running, Role: LEADER
I20251209 20:55:40.003141 13580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:40.004699 13585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.005002 13585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:40.005623 13587 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:40.007967 13587 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:40.008385 13587 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:40.008679 13587 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:40.008985 13587 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:40.009260 13587 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:40.009545 13587 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:40.009809 13587 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:40.010094 13587 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:40.010370 13587 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:40.012254 13587 tablet_loader.cc:96] loaded metadata for tablet 0e200643b18b43dd951a9af667bf1dca (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:40.012548 13587 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:40.012805 13587 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:40.013036 13587 tablet_loader.cc:96] loaded metadata for tablet 2f24e3afd3e24619a10ba98c02ae0899 (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:40.013427 13587 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:40.013638 13587 tablet_loader.cc:96] loaded metadata for tablet 73f0546be0c64a6287404fd0f3a9d427 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:40.014007 13587 tablet_loader.cc:96] loaded metadata for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:40.014427 13587 tablet_loader.cc:96] loaded metadata for tablet c30b153a590a4fc488749b55b2e6ed29 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:40.014705 13587 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:40.014909 13587 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:40.016090 13587 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:40.016212 13587 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:40.017495 13587 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:40.018362 13587 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Loaded TSK: 0
I20251209 20:55:40.019188 13587 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:40.020078 13587 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:40.020886 13587 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:40.099753 13214 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51928:
name: "default.Table_a4904681b195452fb7adf16df235f461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:40.100025 13213 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:40.100343 13212 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e200643b18b43dd951a9af667bf1dca from {username='slave'} at 127.0.0.1:51928
I20251209 20:55:40.100682 13211 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a2925610d5e4948a0c2d93dd5072a2d from {username='slave'} at 127.0.0.1:51928
I20251209 20:55:40.101086 13209 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:40.101332 13209 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:40.101495 13210 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51928:
name: "default.Table_a5e0fd74f125440aba983c391b8177ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:40.101821 13207 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:40.102049 13207 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:40.102216 13208 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51928:
name: "default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:40.100354 13213 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:40.106199 13206 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51928:
name: "default.Table_3d73b33061d7464da6fc7c9367ad163b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:40.107187 13211 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef" } new_table_name: "default.Table_5d01adbefb7248b1a99539733a5c4687" modify_external_catalogs: true
I20251209 20:55:40.107328 13205 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:40.107556 13205 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
F20251209 20:55:40.107775 13268 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f8fd6169dcd google::LogMessage::Fail() at ??:0
I20251209 20:55:40.112116 13518 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Refusing update from remote peer 752a47ea6a344edba26b69df6ea3d482: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
W20251209 20:55:40.112497 13199 consensus_peers.cc:597] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 -> Peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Couldn't send request to peer 1cc9184e29d6430ca18a3fe7984c725c. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:40.112962 13585 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 36, Time since last communication: 0.000s
@ 0x7f8fd616db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8fd61697cc google::LogMessage::Flush() at ??:0
@ 0x7f8fd6169a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f8fecb30aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251209 20:55:40.130283 13574 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.130807 13574 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:40.130533 13596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.131003 13596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:40.129407 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.131183 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:40.132015 13592 mvcc.cc:204] Tried to move back new op lower bound from 7230725079490109440 to 7230725079056117760. Current Snapshot: MvccSnapshot[applied={T|T < 7230725079490109440}]
I20251209 20:55:40.133365 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.133559 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f8fecb2f351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251209 20:55:40.135515 13212 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet 0e200643b18b43dd951a9af667bf1dca
I20251209 20:55:40.139379 13212 catalog_manager.cc:6724] ReplaceTablet: tablet 0e200643b18b43dd951a9af667bf1dca deleted and replaced by tablet 34b3fe48f1d640a684f3aeb35586b354
@ 0x7f8fecb36fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251209 20:55:40.158330 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e200643b18b43dd951a9af667bf1dca with delete_type TABLET_DATA_DELETED (replaced by tablet 34b3fe48f1d640a684f3aeb35586b354) from {username='slave'} at 127.0.0.1:37504
I20251209 20:55:40.158998 13606 tablet_replica.cc:333] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:40.159222 13606 raft_consensus.cc:2243] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:40.159485 13606 raft_consensus.cc:2272] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:40.160436 13606 ts_tablet_manager.cc:1916] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:40.161218 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e200643b18b43dd951a9af667bf1dca with delete_type TABLET_DATA_DELETED (replaced by tablet 34b3fe48f1d640a684f3aeb35586b354) from {username='slave'} at 127.0.0.1:48212
I20251209 20:55:40.163764 13609 tablet_replica.cc:333] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:40.163945 13609 raft_consensus.cc:2243] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:40.164424 13609 raft_consensus.cc:2272] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:40.164999 13609 ts_tablet_manager.cc:1916] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:40.167614 13606 ts_tablet_manager.cc:1929] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:40.167620 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e200643b18b43dd951a9af667bf1dca with delete_type TABLET_DATA_DELETED (replaced by tablet 34b3fe48f1d640a684f3aeb35586b354) from {username='slave'} at 127.0.0.1:36412
I20251209 20:55:40.167749 13606 log.cc:1199] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/0e200643b18b43dd951a9af667bf1dca
I20251209 20:55:40.168666 13212 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51928:
table { table_name: "default.Table_27e25321226d4880ad3b53e9936fd7a8" } new_table_name: "default.Table_855e1033632d4d57b523f0e18d938547" modify_external_catalogs: true
@ 0x7f8fecb36e49 std::_Function_handler<>::_M_invoke() at ??:0
I20251209 20:55:40.171320 13580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:40.171605 13580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
@ 0xa8970d std::function<>::operator()() at ??:?
I20251209 20:55:40.173997 13606 ts_tablet_manager.cc:1950] T 0e200643b18b43dd951a9af667bf1dca P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:40.175230 13198 catalog_manager.cc:4985] TS 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287): tablet 0e200643b18b43dd951a9af667bf1dca (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]) successfully deleted
@ 0x7f8fd8ede3f9 kudu::Thread::SuperviseThread() at ??:0
I20251209 20:55:40.179128 13610 tablet_replica.cc:333] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:40.179322 13610 raft_consensus.cc:2243] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:40.179543 13610 raft_consensus.cc:2272] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:40.179518 12378 tablet_service.cc:1505] Processing CreateTablet for tablet 34b3fe48f1d640a684f3aeb35586b354 (DEFAULT_TABLE table=default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:40.180171 13610 ts_tablet_manager.cc:1916] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:40.180541 12517 tablet_service.cc:1505] Processing CreateTablet for tablet 34b3fe48f1d640a684f3aeb35586b354 (DEFAULT_TABLE table=default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f8fdc4e96db start_thread at ??:0
I20251209 20:55:40.180923 12517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3fe48f1d640a684f3aeb35586b354. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8fd501a71f clone at ??:0
I20251209 20:55:40.183874 12378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3fe48f1d640a684f3aeb35586b354. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:40.187197 13612 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:40.187333 13609 ts_tablet_manager.cc:1929] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:40.187454 13609 log.cc:1199] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/0e200643b18b43dd951a9af667bf1dca
I20251209 20:55:40.187988 13609 ts_tablet_manager.cc:1950] T 0e200643b18b43dd951a9af667bf1dca P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:40.189044 13612 tablet_bootstrap.cc:654] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:40.191021 13613 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:40.192317 13610 ts_tablet_manager.cc:1929] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:40.192448 13610 log.cc:1199] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/0e200643b18b43dd951a9af667bf1dca
I20251209 20:55:40.192767 13613 tablet_bootstrap.cc:654] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:40.192914 13610 ts_tablet_manager.cc:1950] T 0e200643b18b43dd951a9af667bf1dca P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:40.197340 13612 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:40.197474 13612 ts_tablet_manager.cc:1403] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20251209 20:55:40.198112 13613 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:40.198246 13613 ts_tablet_manager.cc:1403] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251209 20:55:40.198275 13612 raft_consensus.cc:359] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.198474 13612 raft_consensus.cc:385] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:40.198552 13612 raft_consensus.cc:740] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:40.198794 13612 consensus_queue.cc:260] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.199322 13612 ts_tablet_manager.cc:1434] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251209 20:55:40.200214 13613 raft_consensus.cc:359] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.200412 13613 raft_consensus.cc:385] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:40.200502 13613 raft_consensus.cc:740] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:40.200739 13613 consensus_queue.cc:260] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.202412 13613 ts_tablet_manager.cc:1434] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251209 20:55:40.355719 12469 connection.cc:537] server connection from 127.0.0.1:37504 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.355717 12608 connection.cc:537] server connection from 127.0.0.1:36412 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.355729 12335 connection.cc:537] server connection from 127.0.0.1:48212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.356145 12333 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.356484 12470 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.357156 12725 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:40.358239 12725 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.252:42679)
W20251209 20:55:40.358517 12725 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:40.359115 12725 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:40.359634 12725 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.252:42679)
I20251209 20:55:40.384619 13297 raft_consensus.cc:493] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:40.384753 13297 raft_consensus.cc:515] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.385587 13297 leader_election.cc:290] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:40.385998 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34b3fe48f1d640a684f3aeb35586b354" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:40.386234 12399 raft_consensus.cc:2468] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:40.386838 12470 leader_election.cc:304] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
W20251209 20:55:40.387054 12472 leader_election.cc:343] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354
I20251209 20:55:40.387156 13297 raft_consensus.cc:2804] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:40.387288 13297 raft_consensus.cc:493] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:40.387369 13297 raft_consensus.cc:3060] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:40.389146 13297 raft_consensus.cc:515] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.389662 13297 leader_election.cc:290] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:40.390005 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34b3fe48f1d640a684f3aeb35586b354" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:40.390158 12399 raft_consensus.cc:3060] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
W20251209 20:55:40.390370 12472 leader_election.cc:343] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354
I20251209 20:55:40.391997 12399 raft_consensus.cc:2468] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:40.392342 12470 leader_election.cc:304] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters: 157c709733f1484799756549aa456c5f
I20251209 20:55:40.392592 13297 raft_consensus.cc:2804] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:40.392939 13297 raft_consensus.cc:697] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:40.393324 13297 consensus_queue.cc:237] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:40.678817 13623 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:40.679004 13623 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:40.679723 13623 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:40.681746 13493 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:40.681944 13493 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:40.682065 13493 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:40.682307 13623 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251209 20:55:40.906908 11795 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu pid 13472
I20251209 20:55:40.913616 13538 generic_service.cc:196] Checking for leaks (request via RPC)
I20251209 20:55:41.029537 11795 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu with pid 13472
I20251209 20:55:41.270712 13297 consensus_queue.cc:1048] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251209 20:55:41.277555 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:41.891952 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--webserver_interface=127.11.132.254
--webserver_port=39627
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:42.210776 13644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:42.211098 13644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:42.211241 13644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:42.219309 13644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:42.219384 13644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:42.219455 13644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:42.219524 13644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:42.229326 13644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.254:40601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.132.254
--webserver_port=39627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:42.230855 13644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:42.232542 13644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:42.239737 13650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:42.239956 13652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:42.239774 13649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:42.240906 13644 server_base.cc:1047] running on GCE node
I20251209 20:55:42.241631 13644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:42.243060 13644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:42.244283 13644 hybrid_clock.cc:648] HybridClock initialized: now 1765313742244235 us; error 33 us; skew 500 ppm
I20251209 20:55:42.244621 13644 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:42.246840 13644 webserver.cc:492] Webserver started at http://127.11.132.254:39627/ using document root <none> and password file <none>
I20251209 20:55:42.247541 13644 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:42.247689 13644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:42.252568 13644 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:42.255030 13658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:42.256188 13644 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 20:55:42.256332 13644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
uuid: "ec484016839448df924d26e6175a3176"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:42.256943 13644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:42.269980 13644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:42.270684 13644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:42.270924 13644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:42.292037 13644 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.254:40601
I20251209 20:55:42.292066 13713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.254:40601 every 8 connection(s)
I20251209 20:55:42.293550 13644 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-0/data/info.pb
I20251209 20:55:42.299153 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13644
I20251209 20:55:42.299243 13714 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:42.303136 13714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap starting.
I20251209 20:55:42.317852 13714 log.cc:826] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:42.345824 13720 ts_tablet_manager.cc:933] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Initiating tablet copy from peer 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:42.346577 13720 tablet_copy_client.cc:323] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.194:41287
I20251209 20:55:42.347419 12557 tablet_copy_service.cc:140] P 3cb2248952314571831b6958a0f0a7b4: Received BeginTabletCopySession request for tablet 34b3fe48f1d640a684f3aeb35586b354 from peer 157c709733f1484799756549aa456c5f ({username='slave'} at 127.11.132.195:34435)
I20251209 20:55:42.347584 12557 tablet_copy_service.cc:161] P 3cb2248952314571831b6958a0f0a7b4: Beginning new tablet copy session on tablet 34b3fe48f1d640a684f3aeb35586b354 from peer 157c709733f1484799756549aa456c5f at {username='slave'} at 127.11.132.195:34435: session id = 157c709733f1484799756549aa456c5f-34b3fe48f1d640a684f3aeb35586b354
I20251209 20:55:42.349119 13714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=26 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:42.349594 13714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Bootstrap complete.
I20251209 20:55:42.349905 12557 tablet_copy_source_session.cc:215] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4: Tablet Copy: opened 0 blocks and 1 log segments
I20251209 20:55:42.351680 13720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b3fe48f1d640a684f3aeb35586b354. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:42.354615 13714 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:42.355258 13714 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec484016839448df924d26e6175a3176, State: Initialized, Role: FOLLOWER
I20251209 20:55:42.355891 13720 tablet_copy_client.cc:806] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: tablet copy: Starting download of 0 data blocks...
I20251209 20:55:42.355851 13714 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:42.356141 13720 tablet_copy_client.cc:670] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: tablet copy: Starting download of 1 WAL segments...
I20251209 20:55:42.356703 13722 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:42.357008 13722 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:42.357599 13714 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:42.358255 13720 tablet_copy_client.cc:538] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251209 20:55:42.362612 13720 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:42.369261 12718 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:42.378878 13737 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:42.379004 13737 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: loading cluster ID for follower catalog manager: success
I20251209 20:55:42.380939 13720 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:42.381407 13720 tablet_bootstrap.cc:492] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Bootstrap complete.
I20251209 20:55:42.381647 13720 ts_tablet_manager.cc:1403] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20251209 20:55:42.382486 13720 raft_consensus.cc:359] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:42.382674 13720 raft_consensus.cc:740] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:42.382917 13720 consensus_queue.cc:260] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:42.383514 13720 ts_tablet_manager.cc:1434] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251209 20:55:42.384557 12557 tablet_copy_service.cc:342] P 3cb2248952314571831b6958a0f0a7b4: Request end of tablet copy session 157c709733f1484799756549aa456c5f-34b3fe48f1d640a684f3aeb35586b354 received from {username='slave'} at 127.11.132.195:34435
I20251209 20:55:42.384744 12557 tablet_copy_service.cc:434] P 3cb2248952314571831b6958a0f0a7b4: ending tablet copy session 157c709733f1484799756549aa456c5f-34b3fe48f1d640a684f3aeb35586b354 on tablet 34b3fe48f1d640a684f3aeb35586b354 with peer 157c709733f1484799756549aa456c5f
I20251209 20:55:42.387178 13675 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:59139; Asking this server to re-register.
I20251209 20:55:42.388232 13737 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: acquiring CA information for follower catalog manager: success
I20251209 20:55:42.389495 12718 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:42.391050 13737 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:42.392040 13675 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:42.407423 13644 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:42.410714 13644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:42.786594 12445 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:42.788336 13675 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:54565; Asking this server to re-register.
I20251209 20:55:42.788959 12445 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:42.790316 13675 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:42.792002 12583 heartbeater.cc:344] Connected to a master server at 127.11.132.254:40601
I20251209 20:55:42.793298 13675 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:41913; Asking this server to re-register.
I20251209 20:55:42.793774 12583 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:42.794806 13675 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:42.846698 13722 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:42.846930 13722 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:42.848721 13722 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:42.849481 13660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:42.851099 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:42.851322 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:42.851421 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:42.851795 13722 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251209 20:55:43.375461 13749 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:43.375649 13749 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:43.376271 13749 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:43.378347 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:43.378497 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:43.378588 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:43.378826 13749 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251209 20:55:43.538735 12672 raft_consensus.cc:1217] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
I20251209 20:55:44.060451 13762 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:44.060624 13762 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:44.061143 13762 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:44.062951 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:44.063205 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:44.063289 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:44.063531 13762 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251209 20:55:45.217105 13776 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:45.217288 13776 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:45.217974 13776 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:45.220443 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:45.220602 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:45.220685 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:45.220899 13776 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251209 20:55:45.412680 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=42787
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:45.722226 13781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:45.722529 13781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:45.722653 13781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:45.730811 13781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:45.730893 13781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:45.730947 13781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:45.730988 13781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:45.740911 13781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=42787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:45.742496 13781 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:45.744293 13781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:45.752229 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:45.752178 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:45.752238 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:45.753283 13781 server_base.cc:1047] running on GCE node
I20251209 20:55:45.754011 13781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:45.755328 13781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:45.756515 13781 hybrid_clock.cc:648] HybridClock initialized: now 1765313745756451 us; error 57 us; skew 500 ppm
I20251209 20:55:45.756845 13781 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:45.759351 13781 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:45.760066 13781 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:45.760178 13781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:45.765385 13781 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 20:55:45.768005 13799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:45.769202 13781 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251209 20:55:45.769359 13781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:45.770008 13781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:45.791919 13781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:45.792825 13781 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:45.793116 13781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:45.812407 13781 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:45.812438 13851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:45.814064 13781 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:45.819859 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13781
I20251209 20:55:45.820209 13852 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:45.823506 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:45.834174 13852 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:45.863487 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=26 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:45.864114 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap complete.
I20251209 20:55:45.869459 13852 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:45.870126 13852 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:45.870752 13852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:45.871491 13856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:45.871752 13856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:45.872567 13852 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:45.889549 13870 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:45.889739 13870 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: success
I20251209 20:55:45.894315 13870 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: acquiring CA information for follower catalog manager: success
I20251209 20:55:45.897392 13870 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:45.902628 13781 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:45.908314 13781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:46.057070 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:46.060165 13817 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:48053; Asking this server to re-register.
I20251209 20:55:46.060890 12582 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:46.062625 13817 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:46.063740 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:46.065088 13817 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:58491; Asking this server to re-register.
I20251209 20:55:46.065618 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:46.066883 13817 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:46.284281 13856 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:46.284554 13856 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:46.286257 13856 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
W20251209 20:55:46.286952 13801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:46.286999 13689 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:46.287436 13689 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 5.
I20251209 20:55:46.288102 13800 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters:
W20251209 20:55:46.288436 13801 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:46.288440 13856 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251209 20:55:46.288606 13856 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:46.288705 13856 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 5 FOLLOWER]: Advancing to term 6
I20251209 20:55:46.292135 13856 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:46.292725 13856 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:46.293136 13689 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176"
I20251209 20:55:46.293341 13689 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 5 FOLLOWER]: Advancing to term 6
W20251209 20:55:46.295059 13801 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:46.297096 13689 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 6.
I20251209 20:55:46.297524 13800 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c
I20251209 20:55:46.298075 13856 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Leader election won for term 6
I20251209 20:55:46.298899 13856 raft_consensus.cc:697] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 LEADER]: Becoming Leader. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Running, Role: LEADER
I20251209 20:55:46.299445 13856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:46.302287 13881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 6 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:46.302558 13881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:46.305383 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:46.305644 13883 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:46.306679 13817 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:49635; Asking this server to re-register.
I20251209 20:55:46.307268 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:46.308507 13883 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:46.308561 13817 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:46.308923 13883 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:46.309291 13883 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:46.309620 13883 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:46.309917 13883 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:46.310230 13883 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:46.310530 13883 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:46.310834 13883 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:46.311146 13883 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:46.313050 13883 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:46.313301 13883 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:46.313565 13883 tablet_loader.cc:96] loaded metadata for tablet 2f24e3afd3e24619a10ba98c02ae0899 (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:46.313783 13883 tablet_loader.cc:96] loaded metadata for tablet 34b3fe48f1d640a684f3aeb35586b354 (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:46.314147 13883 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:46.314356 13883 tablet_loader.cc:96] loaded metadata for tablet 73f0546be0c64a6287404fd0f3a9d427 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:46.314735 13883 tablet_loader.cc:96] loaded metadata for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:46.315083 13883 tablet_loader.cc:96] loaded metadata for tablet c30b153a590a4fc488749b55b2e6ed29 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:46.315308 13883 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:46.315500 13883 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:46.316416 13883 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:46.316486 13883 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:46.317669 13883 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:46.318847 13883 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Loaded TSK: 0
I20251209 20:55:46.319617 13883 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:46.320474 13883 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:46.321301 13883 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:46.336211 13817 master_service.cc:954] ReplaceTablet: received request to replace tablet c30b153a590a4fc488749b55b2e6ed29 from {username='slave'} at 127.0.0.1:57006
I20251209 20:55:46.336912 13815 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_afc836c7c2664ec0bd5f96ce0208fa81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.337469 13809 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_cf01c2d93fcb46cba2bbea670526b180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.337586 13808 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_b7b340bc7e5b46b39332ec8149739649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.338214 13812 master_service.cc:954] ReplaceTablet: received request to replace tablet c30b153a590a4fc488749b55b2e6ed29 from {username='slave'} at 127.0.0.1:57006
I20251209 20:55:46.338085 13814 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_dcb97bd9ec28454eb4f5627100a4e5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.338560 13810 master_service.cc:954] ReplaceTablet: received request to replace tablet 0e200643b18b43dd951a9af667bf1dca from {username='slave'} at 127.0.0.1:57006
I20251209 20:55:46.338748 13813 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_671e465611be47418f159b3c5ee7aa16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.336278 13816 master_service.cc:954] ReplaceTablet: received request to replace tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 from {username='slave'} at 127.0.0.1:57006
I20251209 20:55:46.340363 13810 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:46.340623 13810 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:46.342103 13811 master_service.cc:954] ReplaceTablet: received request to replace tablet 73f0546be0c64a6287404fd0f3a9d427 from {username='slave'} at 127.0.0.1:57006
F20251209 20:55:46.344878 13869 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20251209 20:55:46.346647 13689 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Refusing update from remote peer 752a47ea6a344edba26b69df6ea3d482: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 6 index: 44. (index mismatch)
W20251209 20:55:46.346724 13801 consensus_peers.cc:597] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 -> Peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Couldn't send request to peer 1cc9184e29d6430ca18a3fe7984c725c. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:46.347515 13881 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
@ 0x7fad0c157dcd google::LogMessage::Fail() at ??:0
@ 0x7fad0c15bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fad0c1577cc google::LogMessage::Flush() at ??:0
@ 0x7fad0c157a11 google::LogMessage::~LogMessage() at ??:0
I20251209 20:55:46.356303 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 6 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:46.356575 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fad22b1eaa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251209 20:55:46.357405 13884 mvcc.cc:204] Tried to move back new op lower bound from 7230725105026760704 to 7230725104846516224. Current Snapshot: MvccSnapshot[applied={T|T < 7230725105026760704}]
I20251209 20:55:46.358492 13887 mvcc.cc:204] Tried to move back new op lower bound from 7230725105026760704 to 7230725104846516224. Current Snapshot: MvccSnapshot[applied={T|T < 7230725105026760704}]
@ 0x7fad22b1d351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251209 20:55:46.365481 13816 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845
I20251209 20:55:46.368907 13817 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet c30b153a590a4fc488749b55b2e6ed29
I20251209 20:55:46.369459 13817 catalog_manager.cc:6724] ReplaceTablet: tablet c30b153a590a4fc488749b55b2e6ed29 deleted and replaced by tablet c12a25b37aa14491abdc80b5335914c6
I20251209 20:55:46.372560 13811 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet 73f0546be0c64a6287404fd0f3a9d427
I20251209 20:55:46.373078 13811 catalog_manager.cc:6724] ReplaceTablet: tablet 73f0546be0c64a6287404fd0f3a9d427 deleted and replaced by tablet 9f6c816eae2b42dbb81a5053e22c8b64
I20251209 20:55:46.374562 13816 catalog_manager.cc:6724] ReplaceTablet: tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 deleted and replaced by tablet e31b692527c64e7389889b466bf0ebae
@ 0x7fad22b24fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251209 20:55:46.386778 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:46.386938 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fad22b24e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
I20251209 20:55:46.393282 13816 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.393798 13881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:46.394012 13881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:46.394496 13812 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_a4904681b195452fb7adf16df235f461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.394971 13817 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "default.Table_a5e0fd74f125440aba983c391b8177ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:46.395430 13811 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:46.395673 13811 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
@ 0x7fad0eecc3f9 kudu::Thread::SuperviseThread() at ??:0
I20251209 20:55:46.399170 12517 tablet_service.cc:1552] Processing DeleteTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 with delete_type TABLET_DATA_DELETED (replaced by tablet e31b692527c64e7389889b466bf0ebae) from {username='slave'} at 127.0.0.1:54610
I20251209 20:55:46.399513 12516 tablet_service.cc:1552] Processing DeleteTablet for tablet c30b153a590a4fc488749b55b2e6ed29 with delete_type TABLET_DATA_DELETED (replaced by tablet c12a25b37aa14491abdc80b5335914c6) from {username='slave'} at 127.0.0.1:54610
I20251209 20:55:46.399797 12511 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6c816eae2b42dbb81a5053e22c8b64) from {username='slave'} at 127.0.0.1:54610
I20251209 20:55:46.400691 12651 tablet_service.cc:1552] Processing DeleteTablet for tablet c30b153a590a4fc488749b55b2e6ed29 with delete_type TABLET_DATA_DELETED (replaced by tablet c12a25b37aa14491abdc80b5335914c6) from {username='slave'} at 127.0.0.1:32908
I20251209 20:55:46.400820 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 with delete_type TABLET_DATA_DELETED (replaced by tablet e31b692527c64e7389889b466bf0ebae) from {username='slave'} at 127.0.0.1:32908
I20251209 20:55:46.400944 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 with delete_type TABLET_DATA_DELETED (replaced by tablet e31b692527c64e7389889b466bf0ebae) from {username='slave'} at 127.0.0.1:56128
I20251209 20:55:46.401386 12379 tablet_service.cc:1552] Processing DeleteTablet for tablet c30b153a590a4fc488749b55b2e6ed29 with delete_type TABLET_DATA_DELETED (replaced by tablet c12a25b37aa14491abdc80b5335914c6) from {username='slave'} at 127.0.0.1:56128
I20251209 20:55:46.401612 12377 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6c816eae2b42dbb81a5053e22c8b64) from {username='slave'} at 127.0.0.1:56128
I20251209 20:55:46.401868 12650 tablet_service.cc:1552] Processing DeleteTablet for tablet 73f0546be0c64a6287404fd0f3a9d427 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6c816eae2b42dbb81a5053e22c8b64) from {username='slave'} at 127.0.0.1:32908
I20251209 20:55:46.405962 12377 tablet_service.cc:1505] Processing CreateTablet for tablet 9f6c816eae2b42dbb81a5053e22c8b64 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.406515 12377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6c816eae2b42dbb81a5053e22c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.407512 12652 tablet_service.cc:1505] Processing CreateTablet for tablet e31b692527c64e7389889b466bf0ebae (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.407927 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31b692527c64e7389889b466bf0ebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.411437 12650 tablet_service.cc:1505] Processing CreateTablet for tablet 9f6c816eae2b42dbb81a5053e22c8b64 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.411998 12650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6c816eae2b42dbb81a5053e22c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.416965 12515 tablet_service.cc:1505] Processing CreateTablet for tablet c12a25b37aa14491abdc80b5335914c6 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.417178 12649 tablet_service.cc:1505] Processing CreateTablet for tablet c12a25b37aa14491abdc80b5335914c6 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.417531 12515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a25b37aa14491abdc80b5335914c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.417649 12649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a25b37aa14491abdc80b5335914c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.419904 12516 tablet_service.cc:1505] Processing CreateTablet for tablet e31b692527c64e7389889b466bf0ebae (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.420324 12516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31b692527c64e7389889b466bf0ebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.423923 12511 tablet_service.cc:1505] Processing CreateTablet for tablet 9f6c816eae2b42dbb81a5053e22c8b64 (DEFAULT_TABLE table=default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.424475 12511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6c816eae2b42dbb81a5053e22c8b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.426198 12376 tablet_service.cc:1505] Processing CreateTablet for tablet c12a25b37aa14491abdc80b5335914c6 (DEFAULT_TABLE table=default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.426589 12376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a25b37aa14491abdc80b5335914c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.428084 12379 tablet_service.cc:1505] Processing CreateTablet for tablet e31b692527c64e7389889b466bf0ebae (DEFAULT_TABLE table=default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 20:55:46.428469 12379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31b692527c64e7389889b466bf0ebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 20:55:46.438875 13909 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:46.441347 13909 tablet_bootstrap.cc:654] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.444962 13910 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:46.447423 13910 tablet_bootstrap.cc:654] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.449954 13909 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:46.450140 13909 ts_tablet_manager.cc:1403] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20251209 20:55:46.451051 13909 raft_consensus.cc:359] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.451283 13909 raft_consensus.cc:385] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.451375 13909 raft_consensus.cc:740] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.451673 13909 consensus_queue.cc:260] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.454582 13909 ts_tablet_manager.cc:1434] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251209 20:55:46.455432 13909 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:46.459285 13909 tablet_bootstrap.cc:654] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.461992 13910 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:46.462177 13910 ts_tablet_manager.cc:1403] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
@ 0x7fad124d76db start_thread at ??:0
@ 0x7fad0b00871f clone at ??:0
I20251209 20:55:46.464107 13910 raft_consensus.cc:359] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.465647 13910 raft_consensus.cc:385] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.465759 13910 raft_consensus.cc:740] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.466302 13910 consensus_queue.cc:260] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.467523 13909 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:46.467667 13909 ts_tablet_manager.cc:1403] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20251209 20:55:46.468358 13909 raft_consensus.cc:359] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.468554 13909 raft_consensus.cc:385] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.468631 13909 raft_consensus.cc:740] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.468868 13909 consensus_queue.cc:260] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.470005 13905 tablet_replica.cc:333] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:46.470263 13905 raft_consensus.cc:2243] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.470533 13905 raft_consensus.cc:2272] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.470790 13909 ts_tablet_manager.cc:1434] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251209 20:55:46.471194 13905 ts_tablet_manager.cc:1916] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.471943 13910 ts_tablet_manager.cc:1434] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251209 20:55:46.473807 13909 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4: Bootstrap starting.
I20251209 20:55:46.475361 13909 tablet_bootstrap.cc:654] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.475343 13910 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:46.476240 13903 tablet_replica.cc:333] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:46.476478 13903 raft_consensus.cc:2243] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:46.476946 13903 raft_consensus.cc:2272] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.477205 13910 tablet_bootstrap.cc:654] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.477546 13903 ts_tablet_manager.cc:1916] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.478958 13909 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4: No bootstrap required, opened a new log
I20251209 20:55:46.479100 13909 ts_tablet_manager.cc:1403] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251209 20:55:46.479959 13909 raft_consensus.cc:359] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.480096 13909 raft_consensus.cc:385] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.480190 13909 raft_consensus.cc:740] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.480279 13904 tablet_replica.cc:333] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:46.480414 13909 consensus_queue.cc:260] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.480510 13904 raft_consensus.cc:2243] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.480722 13904 raft_consensus.cc:2272] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.481338 13904 ts_tablet_manager.cc:1916] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.481498 13908 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:46.483464 13908 tablet_bootstrap.cc:654] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.484421 13909 ts_tablet_manager.cc:1434] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251209 20:55:46.488680 13910 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:46.488847 13910 ts_tablet_manager.cc:1403] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20251209 20:55:46.489642 13910 raft_consensus.cc:359] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.489840 13910 raft_consensus.cc:385] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.489928 13910 raft_consensus.cc:740] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.490214 13910 consensus_queue.cc:260] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.491421 13905 ts_tablet_manager.cc:1929] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.491551 13905 log.cc:1199] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/9dfdd1ef1e5a4a60a5c9fcdf9e85a845
I20251209 20:55:46.492063 13905 ts_tablet_manager.cc:1950] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:46.493149 13905 tablet_replica.cc:333] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:46.493384 13905 raft_consensus.cc:2243] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:46.493639 13910 ts_tablet_manager.cc:1434] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251209 20:55:46.493687 13905 raft_consensus.cc:2272] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.493989 13908 tablet_bootstrap.cc:492] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:46.494107 13908 ts_tablet_manager.cc:1403] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20251209 20:55:46.494227 13905 ts_tablet_manager.cc:1916] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.494858 13908 raft_consensus.cc:359] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.495021 13908 raft_consensus.cc:385] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.495103 13908 raft_consensus.cc:740] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.495307 13908 consensus_queue.cc:260] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.496488 13908 ts_tablet_manager.cc:1434] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251209 20:55:46.496755 13908 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:46.498175 13910 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12: Bootstrap starting.
I20251209 20:55:46.498245 13908 tablet_bootstrap.cc:654] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.500162 13910 tablet_bootstrap.cc:654] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.501091 13903 ts_tablet_manager.cc:1929] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.501204 13903 log.cc:1199] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/9dfdd1ef1e5a4a60a5c9fcdf9e85a845
I20251209 20:55:46.501287 13908 tablet_bootstrap.cc:492] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:46.501436 13908 ts_tablet_manager.cc:1403] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251209 20:55:46.501626 13903 ts_tablet_manager.cc:1950] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:46.502031 13908 raft_consensus.cc:359] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.502202 13908 raft_consensus.cc:385] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.502281 13908 raft_consensus.cc:740] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.502496 13903 tablet_replica.cc:333] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:46.502529 13908 consensus_queue.cc:260] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.502676 13903 raft_consensus.cc:2243] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.502947 13908 ts_tablet_manager.cc:1434] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 20:55:46.506433 13908 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f: Bootstrap starting.
I20251209 20:55:46.508188 13908 tablet_bootstrap.cc:654] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f: Neither blocks nor log segments found. Creating new log.
I20251209 20:55:46.506016 13904 ts_tablet_manager.cc:1929] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.508293 13904 log.cc:1199] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/c30b153a590a4fc488749b55b2e6ed29
I20251209 20:55:46.508718 13904 ts_tablet_manager.cc:1950] T c30b153a590a4fc488749b55b2e6ed29 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:46.509109 13910 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12: No bootstrap required, opened a new log
I20251209 20:55:46.509251 13910 ts_tablet_manager.cc:1403] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251209 20:55:46.509238 13903 raft_consensus.cc:2272] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.509630 13904 tablet_replica.cc:333] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:46.509828 13904 raft_consensus.cc:2243] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.509989 13904 raft_consensus.cc:2272] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.509971 13910 raft_consensus.cc:359] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.510097 13908 tablet_bootstrap.cc:492] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f: No bootstrap required, opened a new log
I20251209 20:55:46.510138 13910 raft_consensus.cc:385] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.510219 13908 ts_tablet_manager.cc:1403] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251209 20:55:46.510233 13910 raft_consensus.cc:740] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412a1acaab96447b97a475fb3f94fa12, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.510489 13904 ts_tablet_manager.cc:1916] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.510462 13910 consensus_queue.cc:260] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.510845 13908 raft_consensus.cc:359] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.510998 13908 raft_consensus.cc:385] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 20:55:46.511085 13908 raft_consensus.cc:740] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157c709733f1484799756549aa456c5f, State: Initialized, Role: FOLLOWER
I20251209 20:55:46.511286 13908 consensus_queue.cc:260] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.511755 13908 ts_tablet_manager.cc:1434] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251209 20:55:46.512250 13903 ts_tablet_manager.cc:1916] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.515796 13910 ts_tablet_manager.cc:1434] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251209 20:55:46.518738 13904 ts_tablet_manager.cc:1929] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.518844 13904 log.cc:1199] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/9dfdd1ef1e5a4a60a5c9fcdf9e85a845
I20251209 20:55:46.519192 13904 ts_tablet_manager.cc:1950] T 9dfdd1ef1e5a4a60a5c9fcdf9e85a845 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:46.519994 13904 tablet_replica.cc:333] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:46.520133 13904 raft_consensus.cc:2243] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.520306 13904 raft_consensus.cc:2272] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.520377 13905 ts_tablet_manager.cc:1929] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.520500 13905 log.cc:1199] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/c30b153a590a4fc488749b55b2e6ed29
I20251209 20:55:46.520773 13904 ts_tablet_manager.cc:1916] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.520869 13905 ts_tablet_manager.cc:1950] T c30b153a590a4fc488749b55b2e6ed29 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:46.521721 13905 tablet_replica.cc:333] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:46.521883 13905 raft_consensus.cc:2243] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:46.522040 13905 raft_consensus.cc:2272] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.522504 13905 ts_tablet_manager.cc:1916] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.526369 13904 ts_tablet_manager.cc:1929] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.526469 13904 log.cc:1199] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/73f0546be0c64a6287404fd0f3a9d427
I20251209 20:55:46.526845 13904 ts_tablet_manager.cc:1950] T 73f0546be0c64a6287404fd0f3a9d427 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:46.528527 13905 ts_tablet_manager.cc:1929] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.528638 13905 log.cc:1199] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/73f0546be0c64a6287404fd0f3a9d427
I20251209 20:55:46.529054 13905 ts_tablet_manager.cc:1950] T 73f0546be0c64a6287404fd0f3a9d427 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:46.530121 13903 ts_tablet_manager.cc:1929] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.530248 13903 log.cc:1199] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/c30b153a590a4fc488749b55b2e6ed29
I20251209 20:55:46.530723 13903 ts_tablet_manager.cc:1950] T c30b153a590a4fc488749b55b2e6ed29 P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:46.531694 13903 tablet_replica.cc:333] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: stopping tablet replica
I20251209 20:55:46.531859 13903 raft_consensus.cc:2243] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:46.532191 13903 raft_consensus.cc:2272] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:46.532721 13903 ts_tablet_manager.cc:1916] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:46.538662 13903 ts_tablet_manager.cc:1929] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:46.538761 13903 log.cc:1199] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-1/wal/wals/73f0546be0c64a6287404fd0f3a9d427
I20251209 20:55:46.539124 13903 ts_tablet_manager.cc:1950] T 73f0546be0c64a6287404fd0f3a9d427 P 3cb2248952314571831b6958a0f0a7b4: Deleting consensus metadata
I20251209 20:55:46.582700 13770 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:46.582882 13770 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.583797 13770 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:46.584254 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31b692527c64e7389889b466bf0ebae" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4" is_pre_election: true
I20251209 20:55:46.584444 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31b692527c64e7389889b466bf0ebae" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:46.584502 12536 raft_consensus.cc:2468] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:46.584656 12399 raft_consensus.cc:2468] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 0.
I20251209 20:55:46.584982 12605 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:46.585391 13770 raft_consensus.cc:2804] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:46.585510 13770 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:46.585601 13770 raft_consensus.cc:3060] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.588392 13770 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.589105 13770 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Requested vote from peers 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:46.589546 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31b692527c64e7389889b466bf0ebae" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:46.589676 12536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e31b692527c64e7389889b466bf0ebae" candidate_uuid: "157c709733f1484799756549aa456c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb2248952314571831b6958a0f0a7b4"
I20251209 20:55:46.589737 12399 raft_consensus.cc:3060] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.589869 12536 raft_consensus.cc:3060] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.592463 12399 raft_consensus.cc:2468] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:46.592653 12536 raft_consensus.cc:2468] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157c709733f1484799756549aa456c5f in term 1.
I20251209 20:55:46.592821 12606 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:46.593144 13770 raft_consensus.cc:2804] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:46.593623 13770 raft_consensus.cc:697] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 1 LEADER]: Becoming Leader. State: Replica: 157c709733f1484799756549aa456c5f, State: Running, Role: LEADER
I20251209 20:55:46.593994 13770 consensus_queue.cc:237] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
W20251209 20:55:46.661053 12335 connection.cc:537] server connection from 127.0.0.1:56128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.661073 12608 connection.cc:537] server connection from 127.0.0.1:32908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.661103 12471 connection.cc:537] server connection from 127.0.0.1:54610 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.661105 12333 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.661418 12606 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.662899 12725 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:46.663590 12725 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.252:42679)
W20251209 20:55:46.664181 12725 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:46.664503 12725 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:46.664819 12725 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.252:42679)
I20251209 20:55:46.664865 13755 raft_consensus.cc:493] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:46.665022 13755 raft_consensus.cc:515] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.665655 13755 leader_election.cc:290] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:46.666081 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c12a25b37aa14491abdc80b5335914c6" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:46.666121 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c12a25b37aa14491abdc80b5335914c6" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:46.666276 12672 raft_consensus.cc:2468] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:46.666321 12399 raft_consensus.cc:2468] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:46.666652 12472 leader_election.cc:304] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:46.666872 13755 raft_consensus.cc:2804] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:46.666971 13755 raft_consensus.cc:493] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:46.667057 13755 raft_consensus.cc:3060] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.668807 13755 raft_consensus.cc:515] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.669266 13755 leader_election.cc:290] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:46.669598 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c12a25b37aa14491abdc80b5335914c6" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:46.669672 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c12a25b37aa14491abdc80b5335914c6" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:46.669727 12672 raft_consensus.cc:3060] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.669842 12399 raft_consensus.cc:3060] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.672160 12399 raft_consensus.cc:2468] T c12a25b37aa14491abdc80b5335914c6 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:46.672569 12470 leader_election.cc:304] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:46.672559 12672 raft_consensus.cc:2468] T c12a25b37aa14491abdc80b5335914c6 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:46.672842 13755 raft_consensus.cc:2804] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:46.673251 13755 raft_consensus.cc:697] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:46.673638 13755 consensus_queue.cc:237] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:46.782560 13755 raft_consensus.cc:493] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:46.782744 13755 raft_consensus.cc:515] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.783294 13755 leader_election.cc:290] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:46.783776 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f6c816eae2b42dbb81a5053e22c8b64" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12" is_pre_election: true
I20251209 20:55:46.783788 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f6c816eae2b42dbb81a5053e22c8b64" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f" is_pre_election: true
I20251209 20:55:46.783977 12672 raft_consensus.cc:2468] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:46.783993 12399 raft_consensus.cc:2468] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 0.
I20251209 20:55:46.784349 12472 leader_election.cc:304] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157c709733f1484799756549aa456c5f, 3cb2248952314571831b6958a0f0a7b4; no voters:
I20251209 20:55:46.784663 13755 raft_consensus.cc:2804] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 20:55:46.784761 13755 raft_consensus.cc:493] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:46.784873 13755 raft_consensus.cc:3060] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.786643 13755 raft_consensus.cc:515] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.787169 13755 leader_election.cc:290] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197), 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:46.787495 12399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f6c816eae2b42dbb81a5053e22c8b64" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412a1acaab96447b97a475fb3f94fa12"
I20251209 20:55:46.787657 12672 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f6c816eae2b42dbb81a5053e22c8b64" candidate_uuid: "3cb2248952314571831b6958a0f0a7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157c709733f1484799756549aa456c5f"
I20251209 20:55:46.787719 12399 raft_consensus.cc:3060] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.787860 12672 raft_consensus.cc:3060] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 20:55:46.790373 12399 raft_consensus.cc:2468] T 9f6c816eae2b42dbb81a5053e22c8b64 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:46.790480 12672 raft_consensus.cc:2468] T 9f6c816eae2b42dbb81a5053e22c8b64 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb2248952314571831b6958a0f0a7b4 in term 1.
I20251209 20:55:46.790728 12470 leader_election.cc:304] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4, 412a1acaab96447b97a475fb3f94fa12; no voters:
I20251209 20:55:46.790966 13755 raft_consensus.cc:2804] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 20:55:46.791132 13755 raft_consensus.cc:697] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2248952314571831b6958a0f0a7b4, State: Running, Role: LEADER
I20251209 20:55:46.791460 13755 consensus_queue.cc:237] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } } peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } }
I20251209 20:55:46.904625 13931 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:46.904789 13931 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:46.905356 13931 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:46.907348 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:46.907590 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:46.907687 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:46.907907 13931 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251209 20:55:47.394136 13770 consensus_queue.cc:1048] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:47.400982 13938 consensus_queue.cc:1048] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:47.412770 13644 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:47.412927 13644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:47.442765 13755 consensus_queue.cc:1048] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:47.448817 13755 consensus_queue.cc:1048] T c12a25b37aa14491abdc80b5335914c6 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:47.589442 13954 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:47.589720 13954 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:47.590444 13954 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:47.592155 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:47.592352 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:47.592456 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:47.592671 13954 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251209 20:55:47.755882 13755 consensus_queue.cc:1048] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:47.762737 13951 consensus_queue.cc:1048] T 9f6c816eae2b42dbb81a5053e22c8b64 P 3cb2248952314571831b6958a0f0a7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 20:55:48.499545 13963 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:48.499775 13963 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:48.500382 13963 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:48.501384 13660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20251209 20:55:48.504530 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:48.504745 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:48.504870 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:48.505123 13963 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251209 20:55:49.202553 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--webserver_interface=127.11.132.252
--webserver_port=42787
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251209 20:55:49.512912 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:49.513319 13974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:49.513448 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:49.521644 13974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:49.521715 13974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:49.521800 13974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:49.521857 13974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:49.531123 13974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.252:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.132.252
--webserver_port=42787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:49.532648 13974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:49.534305 13974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:49.541878 13980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:49.541548 13981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:49.542495 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:49.542778 13974 server_base.cc:1047] running on GCE node
I20251209 20:55:49.543444 13974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:49.544653 13974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:49.545819 13974 hybrid_clock.cc:648] HybridClock initialized: now 1765313749545775 us; error 42 us; skew 500 ppm
I20251209 20:55:49.546108 13974 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:49.548065 13974 webserver.cc:492] Webserver started at http://127.11.132.252:42787/ using document root <none> and password file <none>
I20251209 20:55:49.548815 13974 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:49.548930 13974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:49.554039 13974 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20251209 20:55:49.556592 13989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:49.557726 13974 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251209 20:55:49.557868 13974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
uuid: "752a47ea6a344edba26b69df6ea3d482"
format_stamp: "Formatted at 2025-12-09 20:55:21 on dist-test-slave-nk8j"
I20251209 20:55:49.558545 13974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:49.594540 13994 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:49.594702 13994 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:49.595293 13994 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:49.597203 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:49.597376 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:49.597472 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:49.597680 13994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251209 20:55:49.605130 13974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:49.605906 13974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:49.606168 13974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:49.624341 13974 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.252:42679
I20251209 20:55:49.624384 14044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.252:42679 every 8 connection(s)
I20251209 20:55:49.625742 13974 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-2/data/info.pb
I20251209 20:55:49.630684 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 13974
I20251209 20:55:49.632316 14045 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:49.635663 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap starting.
I20251209 20:55:49.650489 14045 log.cc:826] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:49.682755 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap replayed 1/1 log segments. Stats: ops{read=47 overwritten=0 applied=47 ignored=0} inserts{seen=29 ignored=0} mutations{seen=33 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:49.683265 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Bootstrap complete.
I20251209 20:55:49.688261 14045 raft_consensus.cc:359] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:49.689018 14045 raft_consensus.cc:740] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Initialized, Role: FOLLOWER
I20251209 20:55:49.689630 14045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:49.690385 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:49.690629 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:49.691891 14045 sys_catalog.cc:565] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:49.698753 12582 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:49.719406 14066 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:49.719499 14066 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: loading cluster ID for follower catalog manager: success
I20251209 20:55:49.723722 14066 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: acquiring CA information for follower catalog manager: success
I20251209 20:55:49.720999 13974 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:49.727181 13974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:49.727203 14066 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:49.773043 12444 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:49.775173 14010 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:60299; Asking this server to re-register.
I20251209 20:55:49.775813 12444 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:49.777333 14010 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:50.071977 12717 heartbeater.cc:344] Connected to a master server at 127.11.132.252:42679
I20251209 20:55:50.073369 14010 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:32817; Asking this server to re-register.
I20251209 20:55:50.073876 12717 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:50.074913 14010 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:50.081545 14050 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:50.081741 14050 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:50.083642 14050 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:50.084049 13688 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
W20251209 20:55:50.084218 13991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:50.084329 13688 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 6.
I20251209 20:55:50.084908 13990 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters:
I20251209 20:55:50.085283 14050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251209 20:55:50.085418 14050 raft_consensus.cc:493] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 20:55:50.085522 14050 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 6 FOLLOWER]: Advancing to term 7
W20251209 20:55:50.085714 13991 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:50.088855 14050 raft_consensus.cc:515] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:50.089359 14050 leader_election.cc:290] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 election: Requested vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647)
I20251209 20:55:50.089779 13688 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "752a47ea6a344edba26b69df6ea3d482" candidate_term: 7 candidate_status { last_received { term: 6 index: 47 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176"
I20251209 20:55:50.089969 13688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 6 FOLLOWER]: Advancing to term 7
W20251209 20:55:50.090498 13991 leader_election.cc:336] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
I20251209 20:55:50.091972 13688 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752a47ea6a344edba26b69df6ea3d482 in term 7.
I20251209 20:55:50.092286 13990 leader_election.cc:304] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c
I20251209 20:55:50.092561 14050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 7 FOLLOWER]: Leader election won for term 7
I20251209 20:55:50.093020 14050 raft_consensus.cc:697] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [term 7 LEADER]: Becoming Leader. State: Replica: 752a47ea6a344edba26b69df6ea3d482, State: Running, Role: LEADER
I20251209 20:55:50.093431 14050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 6.47, Last appended by leader: 47, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:50.095798 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 7 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:50.096011 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:50.096603 14076 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:50.099151 14076 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:50.099532 14076 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:50.099882 14076 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:50.100198 14076 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:50.100485 14076 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:50.100761 14076 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:50.101009 14076 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:50.101267 14076 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:50.101536 14076 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:50.103331 14076 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:50.103643 14076 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:50.103962 14076 tablet_loader.cc:96] loaded metadata for tablet 2f24e3afd3e24619a10ba98c02ae0899 (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:50.104214 14076 tablet_loader.cc:96] loaded metadata for tablet 34b3fe48f1d640a684f3aeb35586b354 (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:50.104604 14076 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:50.105154 14076 tablet_loader.cc:96] loaded metadata for tablet 9f6c816eae2b42dbb81a5053e22c8b64 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:50.105494 14076 tablet_loader.cc:96] loaded metadata for tablet c12a25b37aa14491abdc80b5335914c6 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:50.105791 14076 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:50.106003 14076 tablet_loader.cc:96] loaded metadata for tablet e31b692527c64e7389889b466bf0ebae (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:50.106171 14076 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:50.106988 14076 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:50.107053 14076 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:50.108232 14076 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:50.109045 14076 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Loaded TSK: 0
I20251209 20:55:50.109719 14076 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:50.110499 14076 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:50.111312 14076 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:50.133359 14010 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f24e3afd3e24619a10ba98c02ae0899 from {username='slave'} at 127.0.0.1:57726
I20251209 20:55:50.133545 14009 master_service.cc:954] ReplaceTablet: received request to replace tablet e31b692527c64e7389889b466bf0ebae from {username='slave'} at 127.0.0.1:57726
I20251209 20:55:50.134132 14006 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f24e3afd3e24619a10ba98c02ae0899 from {username='slave'} at 127.0.0.1:57726
I20251209 20:55:50.134284 14007 master_service.cc:954] ReplaceTablet: received request to replace tablet 34b3fe48f1d640a684f3aeb35586b354 from {username='slave'} at 127.0.0.1:57726
I20251209 20:55:50.134155 14008 master_service.cc:954] ReplaceTablet: received request to replace tablet e31b692527c64e7389889b466bf0ebae from {username='slave'} at 127.0.0.1:57726
I20251209 20:55:50.136536 14005 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_6f837998c1284f3c9172483c60da8813" } new_table_name: "default.Table_581f8285cf154582b0b9bb5096b8ae7a" modify_external_catalogs: true
I20251209 20:55:50.137835 14003 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_e2d4d7aea5fd4d41a0b084f0b7726b94" } new_table_name: "default.Table_a55cc08bf00c49778a226eba00e323f2" modify_external_catalogs: true
I20251209 20:55:50.137360 14004 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_75ebe9a697e04984b340737cc08cb914" } new_table_name: "default.Table_8f50250b772546b68acb72dc72554936" modify_external_catalogs: true
I20251209 20:55:50.137872 14001 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_27e25321226d4880ad3b53e9936fd7a8" } new_table_name: "default.Table_855e1033632d4d57b523f0e18d938547" modify_external_catalogs: true
I20251209 20:55:50.137230 14002 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57726:
name: "default.Table_1b5967177d41405ababac4e174a63fc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:50.144929 13688 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Refusing update from remote peer 752a47ea6a344edba26b69df6ea3d482: Log matching property violated. Preceding OpId in replica: term: 6 index: 47. Preceding OpId from leader: term: 7 index: 49. (index mismatch)
W20251209 20:55:50.145224 13991 consensus_peers.cc:597] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 -> Peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Couldn't send request to peer 1cc9184e29d6430ca18a3fe7984c725c. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:50.145793 14050 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
F20251209 20:55:50.147521 14065 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f8176323dcd google::LogMessage::Fail() at ??:0
@ 0x7f8176327b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f81763237cc google::LogMessage::Flush() at ??:0
@ 0x7f8176323a11 google::LogMessage::~LogMessage() at ??:0
I20251209 20:55:50.159675 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752a47ea6a344edba26b69df6ea3d482. Latest consensus state: current_term: 7 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
@ 0x7f818cceaaa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251209 20:55:50.159894 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:50.161083 14077 mvcc.cc:204] Tried to move back new op lower bound from 7230725120579895296 to 7230725120386011136. Current Snapshot: MvccSnapshot[applied={T|T < 7230725120579895296}]
I20251209 20:55:50.166405 14009 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet e31b692527c64e7389889b466bf0ebae
I20251209 20:55:50.166852 14010 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet 2f24e3afd3e24619a10ba98c02ae0899
W20251209 20:55:50.166942 14009 catalog_manager.cc:5950] Failed to send DeleteReplica request for tablet e31b692527c64e7389889b466bf0ebae: Not found: failed to reset TS proxy: Could not find TS for UUID 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:50.168109 14006 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:50.168313 14006 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_c574530850844bf8b58ae3169558d6b2" } modify_external_catalogs: true
I20251209 20:55:50.168644 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:50.168807 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:50.168761 14008 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57726:
table { table_name: "default.Table_540977ad57d64d6cb18074b1dcbc41ef" } new_table_name: "default.Table_5d01adbefb7248b1a99539733a5c4687" modify_external_catalogs: true
I20251209 20:55:50.168964 14009 catalog_manager.cc:6724] ReplaceTablet: tablet e31b692527c64e7389889b466bf0ebae deleted and replaced by tablet 97bdba5a22254eb89cc36bd02889459e
I20251209 20:55:50.169317 14009 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57726:
name: "default.Table_3d73b33061d7464da6fc7c9367ad163b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 20:55:50.170607 14010 catalog_manager.cc:5950] Failed to send DeleteReplica request for tablet 2f24e3afd3e24619a10ba98c02ae0899: Not found: failed to reset TS proxy: Could not find TS for UUID 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:50.170855 14010 catalog_manager.cc:6724] ReplaceTablet: tablet 2f24e3afd3e24619a10ba98c02ae0899 deleted and replaced by tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff
I20251209 20:55:50.171151 14010 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57726:
name: "default.Table_671e465611be47418f159b3c5ee7aa16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:50.177794 14007 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 752a47ea6a344edba26b69df6ea3d482: Sending DeleteTablet for 3 replicas of tablet 34b3fe48f1d640a684f3aeb35586b354
W20251209 20:55:50.178015 14007 catalog_manager.cc:5950] Failed to send DeleteReplica request for tablet 34b3fe48f1d640a684f3aeb35586b354: Not found: failed to reset TS proxy: Could not find TS for UUID 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:50.178352 14007 catalog_manager.cc:6724] ReplaceTablet: tablet 34b3fe48f1d640a684f3aeb35586b354 deleted and replaced by tablet 7814d6b638964e3b90af1eb85c0a7481
I20251209 20:55:50.178685 14007 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57726:
name: "default.Table_dcb97bd9ec28454eb4f5627100a4e5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20251209 20:55:50.180408 14066 catalog_manager.cc:6277] Error selecting replicas for tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_6f837998c1284f3c9172483c60da8813'. Need at least 3 replicas, but only 2 tablet servers are available
E20251209 20:55:50.180624 14066 catalog_manager.cc:6277] Error selecting replicas for tablet 97bdba5a22254eb89cc36bd02889459e, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_c574530850844bf8b58ae3169558d6b2'. Need at least 3 replicas, but only 2 tablet servers are available
@ 0x7f818cce9351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f818ccf0fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251209 20:55:50.191670 12650 tablet_service.cc:1552] Processing DeleteTablet for tablet 34b3fe48f1d640a684f3aeb35586b354 with delete_type TABLET_DATA_DELETED (replaced by tablet 7814d6b638964e3b90af1eb85c0a7481) from {username='slave'} at 127.0.0.1:32918
I20251209 20:55:50.191762 12651 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f24e3afd3e24619a10ba98c02ae0899 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff) from {username='slave'} at 127.0.0.1:32918
I20251209 20:55:50.192691 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "752a47ea6a344edba26b69df6ea3d482" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:50.192898 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f818ccf0e49 std::_Function_handler<>::_M_invoke() at ??:0
I20251209 20:55:50.191672 12652 tablet_service.cc:1552] Processing DeleteTablet for tablet e31b692527c64e7389889b466bf0ebae with delete_type TABLET_DATA_DELETED (replaced by tablet 97bdba5a22254eb89cc36bd02889459e) from {username='slave'} at 127.0.0.1:32918
W20251209 20:55:50.199316 13991 catalog_manager.cc:4767] Async tablet task DeleteTablet RPC for tablet e31b692527c64e7389889b466bf0ebae on TS 3cb2248952314571831b6958a0f0a7b4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 3cb2248952314571831b6958a0f0a7b4
I20251209 20:55:50.201328 12378 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f24e3afd3e24619a10ba98c02ae0899 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff) from {username='slave'} at 127.0.0.1:56132
I20251209 20:55:50.201534 12376 tablet_service.cc:1552] Processing DeleteTablet for tablet 34b3fe48f1d640a684f3aeb35586b354 with delete_type TABLET_DATA_DELETED (replaced by tablet 7814d6b638964e3b90af1eb85c0a7481) from {username='slave'} at 127.0.0.1:56132
I20251209 20:55:50.201725 12377 tablet_service.cc:1552] Processing DeleteTablet for tablet e31b692527c64e7389889b466bf0ebae with delete_type TABLET_DATA_DELETED (replaced by tablet 97bdba5a22254eb89cc36bd02889459e) from {username='slave'} at 127.0.0.1:56132
I20251209 20:55:50.202117 14094 tablet_replica.cc:333] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:50.202301 14094 raft_consensus.cc:2243] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:50.202476 14094 raft_consensus.cc:2272] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.203364 14094 ts_tablet_manager.cc:1916] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f81790983f9 kudu::Thread::SuperviseThread() at ??:0
I20251209 20:55:50.210315 14094 ts_tablet_manager.cc:1929] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:50.210426 14094 log.cc:1199] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/34b3fe48f1d640a684f3aeb35586b354
I20251209 20:55:50.210860 14094 ts_tablet_manager.cc:1950] T 34b3fe48f1d640a684f3aeb35586b354 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:50.211374 14097 tablet_replica.cc:333] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:50.211549 14097 raft_consensus.cc:2243] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:50.211784 14097 raft_consensus.cc:2272] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.211879 14094 tablet_replica.cc:333] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:50.212041 14094 raft_consensus.cc:2243] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:50.212085 13993 catalog_manager.cc:4985] TS 157c709733f1484799756549aa456c5f (127.11.132.195:38447): tablet 34b3fe48f1d640a684f3aeb35586b354 (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]) successfully deleted
I20251209 20:55:50.212193 14094 raft_consensus.cc:2272] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.212338 14097 ts_tablet_manager.cc:1916] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:50.212687 14094 ts_tablet_manager.cc:1916] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f817c6a36db start_thread at ??:0
E20251209 20:55:50.216955 14066 catalog_manager.cc:6277] Error selecting replicas for tablet 7814d6b638964e3b90af1eb85c0a7481, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_8fbf3f8d855b4f948e57749f9f15b4b4'. Need at least 3 replicas, but only 2 tablet servers are available
E20251209 20:55:50.217084 14066 catalog_manager.cc:6277] Error selecting replicas for tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_6f837998c1284f3c9172483c60da8813'. Need at least 3 replicas, but only 2 tablet servers are available
E20251209 20:55:50.217175 14066 catalog_manager.cc:6277] Error selecting replicas for tablet 97bdba5a22254eb89cc36bd02889459e, reason: Invalid argument: Not enough tablet servers are online for table 'default.table_c574530850844bf8b58ae3169558d6b2'. Need at least 3 replicas, but only 2 tablet servers are available
I20251209 20:55:50.217913 14094 ts_tablet_manager.cc:1929] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
@ 0x7f81751d471f clone at ??:0
I20251209 20:55:50.218011 14094 log.cc:1199] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/2f24e3afd3e24619a10ba98c02ae0899
I20251209 20:55:50.218621 14094 ts_tablet_manager.cc:1950] T 2f24e3afd3e24619a10ba98c02ae0899 P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:50.219512 14094 tablet_replica.cc:333] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: stopping tablet replica
I20251209 20:55:50.219650 14097 ts_tablet_manager.cc:1929] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:50.219707 14094 raft_consensus.cc:2243] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 1 LEADER]: Raft consensus shutting down.
I20251209 20:55:50.219743 14097 log.cc:1199] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/2f24e3afd3e24619a10ba98c02ae0899
I20251209 20:55:50.220042 14097 ts_tablet_manager.cc:1950] T 2f24e3afd3e24619a10ba98c02ae0899 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:50.220182 14094 raft_consensus.cc:2272] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.220750 14097 tablet_replica.cc:333] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:50.220744 14094 ts_tablet_manager.cc:1916] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:50.220894 14097 raft_consensus.cc:2243] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:50.221004 14097 raft_consensus.cc:2272] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.221359 14097 ts_tablet_manager.cc:1916] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:50.226617 14094 ts_tablet_manager.cc:1929] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:50.226729 14094 log.cc:1199] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-2/wal/wals/e31b692527c64e7389889b466bf0ebae
I20251209 20:55:50.227113 14094 ts_tablet_manager.cc:1950] T e31b692527c64e7389889b466bf0ebae P 157c709733f1484799756549aa456c5f: Deleting consensus metadata
I20251209 20:55:50.227490 14097 ts_tablet_manager.cc:1929] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:50.227610 14097 log.cc:1199] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/34b3fe48f1d640a684f3aeb35586b354
I20251209 20:55:50.227995 14097 ts_tablet_manager.cc:1950] T 34b3fe48f1d640a684f3aeb35586b354 P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
I20251209 20:55:50.228629 14097 tablet_replica.cc:333] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: stopping tablet replica
I20251209 20:55:50.228750 14097 raft_consensus.cc:2243] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 20:55:50.228865 14097 raft_consensus.cc:2272] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 20:55:50.229175 14097 ts_tablet_manager.cc:1916] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 20:55:50.233404 14097 ts_tablet_manager.cc:1929] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251209 20:55:50.233484 14097 log.cc:1199] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Deleting WAL directory at /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/ts-0/wal/wals/e31b692527c64e7389889b466bf0ebae
I20251209 20:55:50.233821 14097 ts_tablet_manager.cc:1950] T e31b692527c64e7389889b466bf0ebae P 412a1acaab96447b97a475fb3f94fa12: Deleting consensus metadata
W20251209 20:55:50.258738 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 1: this message will repeat every 5th retry.
W20251209 20:55:50.383671 12605 connection.cc:537] server connection from 127.0.0.1:32918 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:50.383668 12333 connection.cc:537] server connection from 127.0.0.1:56132 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:50.383922 12333 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:50.383968 12725 connection.cc:537] client connection to 127.11.132.252:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:50.384605 12725 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.252:42679)
W20251209 20:55:50.384948 12725 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:50.385155 12725 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:50.385924 12725 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.252:42679)
W20251209 20:55:50.523250 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:50.697191 14104 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:50.697382 14104 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:50.698052 14104 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:50.700225 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:50.700398 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:50.700541 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:50.700886 14104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251209 20:55:50.982136 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:51.232415 14112 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:51.232564 14112 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:51.233131 14112 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:51.235230 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:51.235428 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:51.235512 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:51.235755 14112 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251209 20:55:51.293964 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:52.239362 14121 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:52.239549 14121 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:52.240278 14121 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:52.242841 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647): Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111)
W20251209 20:55:52.243019 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:52.243139 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec484016839448df924d26e6175a3176; no voters: 1cc9184e29d6430ca18a3fe7984c725c, 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:52.243367 14121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251209 20:55:52.414892 13644 master_runner.cc:186] Error getting master registration for 127.11.132.253:34647: Network error: Client connection negotiation failed: client connection to 127.11.132.253:34647: connect: Connection refused (error 111),
I20251209 20:55:52.415036 13644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:52.501261 11795 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
/tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--webserver_interface=127.11.132.253
--webserver_port=34115
--builtin_ntp_servers=127.11.132.212:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251209 20:55:52.551862 14117 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:55:52.552052 14117 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:52.552706 14117 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:55:52.553638 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:55:52.553810 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:55:52.553973 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:55:52.554216 14117 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:55:52.824985 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 20:55:52.825258 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 20:55:52.825328 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 20:55:52.834554 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 20:55:52.834662 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 20:55:52.834744 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 20:55:52.834802 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 20:55:52.844154 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.212:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:41539
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.253:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.132.253
--webserver_port=34115
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type FASTDEBUG
built by None at 09 Dec 2025 20:43:20 UTC on 24a791456cd2
build id 9436
ASAN enabled
I20251209 20:55:52.845518 14126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 20:55:52.846964 14126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 20:55:52.854322 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:52.854501 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 20:55:52.854918 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 20:55:52.855304 14126 server_base.cc:1047] running on GCE node
I20251209 20:55:52.855918 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 20:55:52.857012 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 20:55:52.858191 14126 hybrid_clock.cc:648] HybridClock initialized: now 1765313752858141 us; error 46 us; skew 500 ppm
I20251209 20:55:52.858515 14126 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251209 20:55:52.860518 14126 webserver.cc:492] Webserver started at http://127.11.132.253:34115/ using document root <none> and password file <none>
I20251209 20:55:52.861186 14126 fs_manager.cc:362] Metadata directory not provided
I20251209 20:55:52.861280 14126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 20:55:52.866014 14126 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 20:55:52.868489 14140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 20:55:52.869639 14126 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 20:55:52.869779 14126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data,/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
uuid: "1cc9184e29d6430ca18a3fe7984c725c"
format_stamp: "Formatted at 2025-12-09 20:55:20 on dist-test-slave-nk8j"
I20251209 20:55:52.870397 14126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 20:55:52.884263 14126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 20:55:52.884986 14126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 20:55:52.885241 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 20:55:52.904626 14126 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.253:34647
I20251209 20:55:52.904657 14191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.253:34647 every 8 connection(s)
I20251209 20:55:52.906138 14126 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/master-1/data/info.pb
I20251209 20:55:52.909438 11795 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskd5EcMJ/build/asan/bin/kudu as pid 14126
I20251209 20:55:52.913568 14192 sys_catalog.cc:263] Verifying existing consensus state
I20251209 20:55:52.917133 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap starting.
I20251209 20:55:52.928217 14192 log.cc:826] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Log is configured to *not* fsync() on all Append() calls
I20251209 20:55:52.957089 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=39 ignored=0} inserts{seen=25 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 20:55:52.957664 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: Bootstrap complete.
I20251209 20:55:52.963155 14192 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:52.963886 14192 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9184e29d6430ca18a3fe7984c725c, State: Initialized, Role: FOLLOWER
I20251209 20:55:52.964473 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:52.965190 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:52.965452 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:52.966193 14192 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: configured and running, proceeding with master startup.
I20251209 20:55:52.984544 14210 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:52.984726 14210 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: loading cluster ID for follower catalog manager: success
I20251209 20:55:52.991137 14210 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: acquiring CA information for follower catalog manager: success
I20251209 20:55:52.995044 14210 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 20:55:53.011348 14126 master_runner.cc:186] Error getting master registration for 127.11.132.252:42679: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111),
I20251209 20:55:53.016690 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:53.081822 12719 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:53.084877 14157 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "157c709733f1484799756549aa456c5f" instance_seqno: 1765313722969376) as {username='slave'} at 127.11.132.195:35409; Asking this server to re-register.
I20251209 20:55:53.085556 12719 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:53.087119 14157 ts_manager.cc:194] Registered new tserver with Master: 157c709733f1484799756549aa456c5f (127.11.132.195:38447)
I20251209 20:55:53.130893 12584 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:53.132547 14157 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" instance_seqno: 1765313722556966) as {username='slave'} at 127.11.132.194:46579; Asking this server to re-register.
I20251209 20:55:53.133140 12584 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:53.134294 14157 ts_manager.cc:194] Registered new tserver with Master: 3cb2248952314571831b6958a0f0a7b4 (127.11.132.194:41287)
I20251209 20:55:53.280997 12446 heartbeater.cc:344] Connected to a master server at 127.11.132.253:34647
I20251209 20:55:53.282581 14157 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" instance_seqno: 1765313722109642) as {username='slave'} at 127.11.132.193:33295; Asking this server to re-register.
I20251209 20:55:53.283180 12446 heartbeater.cc:461] Registering TS with master...
I20251209 20:55:53.284534 14157 ts_manager.cc:194] Registered new tserver with Master: 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
I20251209 20:55:53.546119 14217 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 20:55:53.546367 14217 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:53.548094 14217 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:53.548561 13689 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cc9184e29d6430ca18a3fe7984c725c" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: false dest_uuid: "ec484016839448df924d26e6175a3176" is_pre_election: true
I20251209 20:55:53.548822 13689 raft_consensus.cc:2368] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cc9184e29d6430ca18a3fe7984c725c for earlier term 5. Current term is 7.
W20251209 20:55:53.548900 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:53.549502 14141 leader_election.cc:400] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 5 pre-election: Vote denied by peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cc9184e29d6430ca18a3fe7984c725c for earlier term 5. Current term is 7.
I20251209 20:55:53.549592 14141 leader_election.cc:403] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251209 20:55:53.549887 14217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 4 FOLLOWER]: Advancing to term 7
W20251209 20:55:53.550263 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:53.553282 14217 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 7 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cc9184e29d6430ca18a3fe7984c725c for earlier term 5. Current term is 7.
I20251209 20:55:53.686421 14221 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:53.686606 14221 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:53.687132 14221 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:53.688633 13660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251209 20:55:53.689821 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:53.692898 14167 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 8 candidate_status { last_received { term: 7 index: 52 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c" is_pre_election: true
I20251209 20:55:53.693292 14167 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec484016839448df924d26e6175a3176 in term 7.
I20251209 20:55:53.693698 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176; no voters: 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:53.693964 14221 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251209 20:55:53.694063 14221 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 752a47ea6a344edba26b69df6ea3d482)
I20251209 20:55:53.694178 14221 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 7 FOLLOWER]: Advancing to term 8
I20251209 20:55:53.696059 14221 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:53.696573 14221 leader_election.cc:290] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 election: Requested vote from peers 1cc9184e29d6430ca18a3fe7984c725c (127.11.132.253:34647), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
I20251209 20:55:53.696939 14167 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec484016839448df924d26e6175a3176" candidate_term: 8 candidate_status { last_received { term: 7 index: 52 } } ignore_live_leader: false dest_uuid: "1cc9184e29d6430ca18a3fe7984c725c"
I20251209 20:55:53.697120 14167 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 7 FOLLOWER]: Advancing to term 8
W20251209 20:55:53.697764 13660 leader_election.cc:336] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:53.699358 14167 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec484016839448df924d26e6175a3176 in term 8.
I20251209 20:55:53.699728 13660 leader_election.cc:304] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c, ec484016839448df924d26e6175a3176; no voters: 752a47ea6a344edba26b69df6ea3d482
I20251209 20:55:53.699976 14221 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 8 FOLLOWER]: Leader election won for term 8
I20251209 20:55:53.700376 14221 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [term 8 LEADER]: Becoming Leader. State: Replica: ec484016839448df924d26e6175a3176, State: Running, Role: LEADER
I20251209 20:55:53.700763 14221 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 7.52, Last appended by leader: 52, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:53.702332 14225 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec484016839448df924d26e6175a3176. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.702639 14225 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: LEADER
I20251209 20:55:53.703248 14227 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 20:55:53.705721 14227 catalog_manager.cc:679] Loaded metadata for table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703]
I20251209 20:55:53.706154 14227 catalog_manager.cc:679] Loaded metadata for table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63]
I20251209 20:55:53.706496 14227 catalog_manager.cc:679] Loaded metadata for table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675]
I20251209 20:55:53.706815 14227 catalog_manager.cc:679] Loaded metadata for table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7]
I20251209 20:55:53.707109 14227 catalog_manager.cc:679] Loaded metadata for table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d]
I20251209 20:55:53.707424 14227 catalog_manager.cc:679] Loaded metadata for table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6]
I20251209 20:55:53.707790 14227 catalog_manager.cc:679] Loaded metadata for table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a]
I20251209 20:55:53.708134 14227 catalog_manager.cc:679] Loaded metadata for table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5]
I20251209 20:55:53.708458 14227 catalog_manager.cc:679] Loaded metadata for table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa]
I20251209 20:55:53.710345 14227 tablet_loader.cc:96] loaded metadata for tablet 0f187335da3a4b39b4c453eb470e99e8 (table default.table_27e25321226d4880ad3b53e9936fd7a8 [id=144211a70b224385a06cf126659b9675])
I20251209 20:55:53.710680 14227 tablet_loader.cc:96] loaded metadata for tablet 19c5cdf7ec11442e99363cb062365d38 (table default.table_08951a6a698b49d0b1229aa3bebf4877 [id=ecee1a8f8e254871b7a97b06f6d3bafa])
I20251209 20:55:53.711364 14227 tablet_loader.cc:96] loaded metadata for tablet 73119767050448079a74b823ea6587f5 (table default.table_e2d4d7aea5fd4d41a0b084f0b7726b94 [id=e521de72de6548b9a3f40230f8948cd5])
I20251209 20:55:53.711833 14227 tablet_loader.cc:96] loaded metadata for tablet 7814d6b638964e3b90af1eb85c0a7481 (table default.table_8fbf3f8d855b4f948e57749f9f15b4b4 [id=e1af992dce454194bee928a8b9314f3a])
I20251209 20:55:53.712260 14227 tablet_loader.cc:96] loaded metadata for tablet 8e0ae55ef5c84ba3b4e6d93bbe3c6eff (table default.table_6f837998c1284f3c9172483c60da8813 [id=0db2a490be97445e8785a588de473e63])
I20251209 20:55:53.712522 14227 tablet_loader.cc:96] loaded metadata for tablet 97bdba5a22254eb89cc36bd02889459e (table default.table_c574530850844bf8b58ae3169558d6b2 [id=1aa7c64fc80b4190af0a745c7615e6e7])
I20251209 20:55:53.712966 14227 tablet_loader.cc:96] loaded metadata for tablet 9f6c816eae2b42dbb81a5053e22c8b64 (table default.table_25ee0c76d07240769b864a5048b8db94 [id=0c704d87ded842a3baf0944aee2b9703])
I20251209 20:55:53.713423 14227 tablet_loader.cc:96] loaded metadata for tablet c12a25b37aa14491abdc80b5335914c6 (table default.table_540977ad57d64d6cb18074b1dcbc41ef [id=dcac7cc968d943ce87f66a5f4e3fb4e6])
I20251209 20:55:53.713872 14227 tablet_loader.cc:96] loaded metadata for tablet cb8c74dcd4374a0fa71a259d5961f7c4 (table default.table_75ebe9a697e04984b340737cc08cb914 [id=492e13897e504ce7a9c2d22a1a112f3d])
I20251209 20:55:53.714277 14227 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 20:55:53.715245 14227 catalog_manager.cc:1269] Loaded cluster ID: 0626cb9ccf5a4e969f738e77db5b55c2
I20251209 20:55:53.715327 14227 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 20:55:53.716637 14227 catalog_manager.cc:1514] Loading token signing keys...
I20251209 20:55:53.717564 14227 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176: Loaded TSK: 0
I20251209 20:55:53.718325 14227 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 20:55:53.719234 14227 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251209 20:55:53.720359 14227 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251209 20:55:53.772647 13674 master_service.cc:954] ReplaceTablet: received request to replace tablet 73119767050448079a74b823ea6587f5 from {username='slave'} at 127.0.0.1:35112
I20251209 20:55:53.773743 13673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:53.773550 13672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:53.774030 13673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_25ee0c76d07240769b864a5048b8db94" } modify_external_catalogs: true
I20251209 20:55:53.774616 13666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "default.Table_cf01c2d93fcb46cba2bbea670526b180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:53.774950 13675 master_service.cc:954] ReplaceTablet: received request to replace tablet 73119767050448079a74b823ea6587f5 from {username='slave'} at 127.0.0.1:35112
I20251209 20:55:53.775286 13667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "default.Table_b7b340bc7e5b46b39332ec8149739649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:53.775919 13668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:53.776188 13668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_08951a6a698b49d0b1229aa3bebf4877" } modify_external_catalogs: true
I20251209 20:55:53.776113 13670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "default.Table_a4904681b195452fb7adf16df235f461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 20:55:53.777086 13671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:53.777318 13671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35112:
table { table_name: "default.Table_8fbf3f8d855b4f948e57749f9f15b4b4" } modify_external_catalogs: true
I20251209 20:55:53.777647 13669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35112:
name: "default.Table_a5e0fd74f125440aba983c391b8177ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 20:55:53.779301 13660 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 -> Peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Couldn't send request to peer 752a47ea6a344edba26b69df6ea3d482. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 20:55:53.779337 14167 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Refusing update from remote peer ec484016839448df924d26e6175a3176: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 8 index: 54. (index mismatch)
I20251209 20:55:53.780117 14225 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 39, Time since last communication: 0.000s
F20251209 20:55:53.781183 13735 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_540977ad57d64d6cb18074b1dcbc41ef, message={"server":"","servicePrincipal":"","db":"default","table":"table_540977ad57d64d6cb18074b1dcbc41ef","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_540977ad57d64d6cb18074b1dcbc41ef\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_540977ad57d64d6cb18074b1dcbc41ef\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_540977ad57d64d6cb18074b1dcbc41ef\",\"kudu.table_id\":\"dcac7cc968d943ce87f66a5f4e3fb4e6\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765313723, eventType=CREATE_TABLE, dbName=default, tableName=table_79b3eb661bba41bb99071683748470b5, message={"server":"","servicePrincipal":"","db":"default","table":"table_79b3eb661bba41bb99071683748470b5","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_79b3eb661bba41bb99071683748470b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765313723},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskd5EcMJ/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765313706309372-11795-0/minicluster-data/warehouse/table_79b3eb661bba41bb99071683748470b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_79b3eb661bba41bb99071683748470b5\",\"kudu.table_id\":\"29a63f9f2b844ee38b56d668765864f9\",\"transient_lastDdlTime\":\"1765313723\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0626cb9ccf5a4e969f738e77db5b55c2\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765313723,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f12a7a83dcd google::LogMessage::Fail() at ??:0
@ 0x7f12a7a87b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f12a7a837cc google::LogMessage::Flush() at ??:0
@ 0x7f12a7a83a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f12be44aaa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f12be449351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f12be450fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f12be450e49 std::_Function_handler<>::_M_invoke() at ??:0
I20251209 20:55:53.807278 14230 mvcc.cc:204] Tried to move back new op lower bound from 7230725120744087552 to 7230725079056117760. Current Snapshot: MvccSnapshot[applied={T|T < 7230725120744087552 or (T in {7230725120744087552})}]
I20251209 20:55:53.808295 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec484016839448df924d26e6175a3176. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.808566 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
@ 0xa8970d std::function<>::operator()() at ??:?
I20251209 20:55:53.808984 14235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.809283 14235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:53.809108 14233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.809473 14233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
I20251209 20:55:53.810712 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.810952 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f12aa7f83f9 kudu::Thread::SuperviseThread() at ??:0
I20251209 20:55:53.813767 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.813941 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f12ade036db start_thread at ??:0
I20251209 20:55:53.816308 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "ec484016839448df924d26e6175a3176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } } }
I20251209 20:55:53.816586 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec484016839448df924d26e6175a3176 [sys.catalog]: This master's current role is: LEADER
@ 0x7f12a693471f clone at ??:0
W20251209 20:55:53.976570 12469 connection.cc:537] client connection to 127.11.132.254:40601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 20:55:53.977676 12724 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.132.254:40601)
W20251209 20:55:53.978382 12724 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.132.254:40601)
W20251209 20:55:53.978880 12724 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.132.254:40601)
W20251209 20:55:53.979238 12724 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.132.254:40601)
I20251209 20:55:54.312291 14240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:54.312464 14240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:54.313169 14240 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:54.315187 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:54.315337 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:54.315490 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:54.315811 14240 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:55:54.994570 14261 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:54.994700 14261 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:54.995213 14261 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:54.997704 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:54.997838 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:54.997944 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:54.998142 14261 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:55:55.097443 11795 master-stress-test.cc:577] Killing test thread 1/25
I20251209 20:55:55.812379 14266 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:55.812582 14266 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:55.813257 14266 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:55.816017 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:55.816555 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:55.816689 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:55.816932 14266 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:55:56.676894 14117 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:55:56.677112 14117 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:55:56.677883 14117 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:55:56.678781 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:55:56.678949 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:55:56.679082 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:55:56.679309 14117 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:55:56.683017 14273 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:56.683204 14273 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:56.683810 14273 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:56.685760 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:55:56.686089 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:55:56.686229 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:56.686479 14273 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:55:58.017990 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:55:58.018168 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:55:58.514665 14284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:55:58.514842 14284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:55:58.515384 14284 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:55:58.516978 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:55:58.517181 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:55:58.517308 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:55:58.517519 14284 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:56:00.151155 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 6: this message will repeat every 5th retry.
W20251209 20:56:00.610705 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 6: this message will repeat every 5th retry.
I20251209 20:56:01.096117 14304 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:01.096340 14304 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:01.097098 14304 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:01.097908 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20251209 20:56:01.099949 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:01.100255 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:01.100425 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:01.100916 14304 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:56:01.312937 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 6: this message will repeat every 5th retry.
W20251209 20:56:01.601869 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 6: this message will repeat every 5th retry.
I20251209 20:56:02.773907 14323 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:56:02.774085 14323 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:56:02.774703 14323 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:56:02.775604 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:56:02.775743 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:56:02.775895 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:56:02.776137 14323 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:56:03.019361 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:03.019587 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:04.506728 14339 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:04.506927 14339 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:04.507536 14339 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:04.509435 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:04.509619 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:04.509716 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:04.509950 14339 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:07.616549 14322 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:56:07.616710 14322 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:56:07.617327 14322 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:56:07.618228 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:56:07.618430 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:56:07.618592 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:56:07.619045 14351 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:56:08.020831 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:08.021004 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:10.502296 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 11: this message will repeat every 5th retry.
W20251209 20:56:10.679045 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 11: this message will repeat every 5th retry.
W20251209 20:56:11.343022 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 11: this message will repeat every 5th retry.
W20251209 20:56:11.525046 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 11: this message will repeat every 5th retry.
I20251209 20:56:11.812564 14396 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:11.812773 14396 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:11.813427 14396 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:11.814270 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251209 20:56:11.815697 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:11.815851 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:11.815980 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:11.816190 14396 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:13.024645 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:13.024857 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:13.151891 14405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:13.152086 14405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:13.152751 14405 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:13.154340 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:13.154505 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:13.154605 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:13.154846 14405 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:18.026319 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:18.026520 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:19.123111 14427 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:56:19.123292 14427 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:56:19.124037 14427 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:56:19.124903 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:56:19.125092 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:56:19.125181 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:56:19.125418 14427 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:56:20.461867 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 16: this message will repeat every 5th retry.
W20251209 20:56:20.678807 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 16: this message will repeat every 5th retry.
W20251209 20:56:20.750634 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 16: this message will repeat every 5th retry.
W20251209 20:56:21.584085 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 16: this message will repeat every 5th retry.
I20251209 20:56:23.028070 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:23.028239 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:23.433986 12583 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (132 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:23.452633 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (66 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:56:23.654309 12717 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (66 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:28.029650 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:28.029918 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:29.826010 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 21: this message will repeat every 5th retry.
I20251209 20:56:29.969792 14467 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:56:29.969975 14467 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:56:29.970557 14467 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:56:29.971341 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:56:29.971508 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:56:29.971652 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:56:29.971877 14467 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:56:31.111660 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 21: this message will repeat every 5th retry.
W20251209 20:56:31.313563 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 21: this message will repeat every 5th retry.
W20251209 20:56:31.368291 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 21: this message will repeat every 5th retry.
I20251209 20:56:31.979497 14489 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:31.979686 14489 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:31.980338 14489 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:31.981050 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251209 20:56:31.982172 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:31.982375 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:31.982466 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:31.982632 14489 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:33.031430 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:33.031649 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:38.033043 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:38.033224 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:40.281840 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 26: this message will repeat every 5th retry.
W20251209 20:56:40.709894 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 26: this message will repeat every 5th retry.
W20251209 20:56:40.757831 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 26: this message will repeat every 5th retry.
W20251209 20:56:40.866391 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 26: this message will repeat every 5th retry.
I20251209 20:56:42.221990 14574 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:42.222223 14574 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:42.222914 14574 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:42.223948 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:56:42.225046 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:56:42.225208 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:56:42.225332 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:42.225574 14574 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:43.034842 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:43.035017 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:47.309026 14620 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:56:47.309214 14620 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:56:47.309971 14620 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:56:47.310856 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:56:47.311009 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:56:47.311100 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:56:47.311378 14620 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:56:48.036523 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:48.036741 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:50.203043 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 31: this message will repeat every 5th retry.
W20251209 20:56:50.348376 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 31: this message will repeat every 5th retry.
W20251209 20:56:50.741658 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 31: this message will repeat every 5th retry.
W20251209 20:56:51.284297 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 31: this message will repeat every 5th retry.
I20251209 20:56:53.037992 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:53.038142 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:56:56.324119 14692 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:56:56.324311 14692 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:56:56.325067 14692 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:56:56.326064 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:56:56.327365 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:56:56.327520 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:56:56.327697 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:56:56.327945 14692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:56:58.039456 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:56:58.039693 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:56:59.903175 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 36: this message will repeat every 5th retry.
W20251209 20:57:00.763654 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 36: this message will repeat every 5th retry.
W20251209 20:57:01.304792 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 36: this message will repeat every 5th retry.
W20251209 20:57:01.708662 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 36: this message will repeat every 5th retry.
I20251209 20:57:03.041009 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:03.041183 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:57:06.713734 14750 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:57:06.713951 14750 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:57:06.714745 14750 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:57:06.715631 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:57:06.715775 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:57:06.715876 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:57:06.716097 14750 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:57:07.210443 14786 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:57:07.210659 14786 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:57:07.211374 14786 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:57:07.212270 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:57:07.213348 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:57:07.213507 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:57:07.213624 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:57:07.213822 14786 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:57:08.042519 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:08.042693 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:09.480165 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 41: this message will repeat every 5th retry.
W20251209 20:57:09.697811 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 41: this message will repeat every 5th retry.
W20251209 20:57:11.064862 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 41: this message will repeat every 5th retry.
W20251209 20:57:11.126612 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 41: this message will repeat every 5th retry.
I20251209 20:57:13.044170 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:13.044333 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:57:13.960742 14834 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:57:13.960929 14834 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:57:13.961484 14834 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:57:13.962337 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:57:13.962503 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:57:13.962623 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:57:13.962811 14834 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:57:14.693552 14848 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:57:14.693768 14848 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:57:14.694501 14848 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:57:14.695489 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:57:14.696544 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:57:14.696682 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:57:14.696803 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:57:14.697018 14848 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:57:18.045760 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:18.045944 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:18.870110 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 46: this message will repeat every 5th retry.
W20251209 20:57:19.732959 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 46: this message will repeat every 5th retry.
W20251209 20:57:20.926796 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 46: this message will repeat every 5th retry.
W20251209 20:57:21.381606 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 46: this message will repeat every 5th retry.
I20251209 20:57:23.047385 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:23.047618 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:23.545331 12583 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (404 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:57:23.580497 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (186 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:57:23.818818 12718 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (179 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:57:28.049026 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:28.049218 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:28.995353 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 51: this message will repeat every 5th retry.
W20251209 20:57:29.817313 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 51: this message will repeat every 5th retry.
W20251209 20:57:30.707953 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 51: this message will repeat every 5th retry.
I20251209 20:57:31.200296 14994 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:57:31.200470 14994 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:57:31.201272 14994 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:57:31.202054 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:57:31.202224 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:57:31.202394 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:57:31.202615 14994 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:57:31.485227 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 51: this message will repeat every 5th retry.
I20251209 20:57:31.559976 15001 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:57:31.560161 15001 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:57:31.560948 15001 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:57:31.561931 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:57:31.563213 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:57:31.563395 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:57:31.563517 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:57:31.563773 15001 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:57:33.050640 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:33.050839 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:57:38.052083 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:38.052260 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:38.730870 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 56: this message will repeat every 5th retry.
W20251209 20:57:39.899446 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 56: this message will repeat every 5th retry.
W20251209 20:57:40.881006 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 56: this message will repeat every 5th retry.
W20251209 20:57:41.324994 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 56: this message will repeat every 5th retry.
I20251209 20:57:43.053781 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:43.053946 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:57:44.693956 15097 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:57:44.694123 15097 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:57:44.694672 15097 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:57:44.695324 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:57:44.695475 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:57:44.695640 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:57:44.695844 15097 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:57:48.055523 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:48.055750 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:48.903297 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 61: this message will repeat every 5th retry.
W20251209 20:57:49.763140 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 61: this message will repeat every 5th retry.
I20251209 20:57:51.233330 15162 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:57:51.233503 15162 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:57:51.234236 15162 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:57:51.235208 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:57:51.236281 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:57:51.236420 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:57:51.236567 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:57:51.236825 15162 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:57:51.254318 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 61: this message will repeat every 5th retry.
W20251209 20:57:51.968720 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 61: this message will repeat every 5th retry.
I20251209 20:57:53.037917 14146 maintenance_manager.cc:419] P 1cc9184e29d6430ca18a3fe7984c725c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033354
I20251209 20:57:53.060510 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:53.060674 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:57:53.075881 14145 maintenance_manager.cc:643] P 1cc9184e29d6430ca18a3fe7984c725c: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.020s sys 0.005s Metrics: {"bytes_written":14709,"cfile_init":1,"dirs.queue_time_us":292,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":724,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":33,"thread_start_us":354,"threads_started":2,"wal-append.queue_time_us":235}
I20251209 20:57:53.076954 14146 maintenance_manager.cc:419] P 1cc9184e29d6430ca18a3fe7984c725c: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1794 bytes on disk
I20251209 20:57:53.077576 14145 maintenance_manager.cc:643] P 1cc9184e29d6430ca18a3fe7984c725c: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20251209 20:57:55.092223 15190 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:57:55.092398 15190 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:57:55.093211 15190 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:57:55.093988 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:57:55.094130 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:57:55.094244 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:57:55.094494 15190 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:57:58.062034 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:57:58.062258 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:57:58.749404 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 66: this message will repeat every 5th retry.
W20251209 20:57:59.325090 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 66: this message will repeat every 5th retry.
W20251209 20:58:01.768420 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 66: this message will repeat every 5th retry.
W20251209 20:58:01.998984 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 66: this message will repeat every 5th retry.
I20251209 20:58:03.063925 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:03.064168 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:58:06.600661 15289 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:58:06.600807 15289 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:58:06.601344 15289 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:58:06.602113 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:58:06.602260 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:58:06.602370 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:58:06.602586 15289 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:58:08.065752 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:08.065973 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:09.275831 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 71: this message will repeat every 5th retry.
W20251209 20:58:09.356206 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 71: this message will repeat every 5th retry.
I20251209 20:58:09.413828 15342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:09.414018 15342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:09.414763 15342 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:09.415679 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:58:09.416769 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:58:09.416946 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:58:09.417050 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:09.417230 15342 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:58:11.334529 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 71: this message will repeat every 5th retry.
W20251209 20:58:12.243525 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 71: this message will repeat every 5th retry.
I20251209 20:58:12.480001 15394 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:12.480193 15394 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:12.480835 15394 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:12.482759 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:58:12.482935 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:58:12.483079 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:12.483328 15394 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:58:13.067625 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:13.067795 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:58:18.069157 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:18.069346 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:18.870534 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 76: this message will repeat every 5th retry.
W20251209 20:58:19.152138 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 76: this message will repeat every 5th retry.
W20251209 20:58:21.036964 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 76: this message will repeat every 5th retry.
I20251209 20:58:22.462177 15514 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:22.462350 15514 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:22.462931 15514 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:22.463855 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251209 20:58:22.464913 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:58:22.465127 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:58:22.465260 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:22.465503 15514 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:58:23.070694 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:23.070905 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:58:23.295125 15508 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:58:23.295315 15508 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:58:23.296172 15508 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:58:23.297204 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:58:23.297374 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:58:23.297482 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:58:23.297700 15508 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:58:23.707260 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (306 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:58:23.768898 12583 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (675 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:58:23.769294 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 76: this message will repeat every 5th retry.
W20251209 20:58:23.983876 12718 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (299 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:58:28.072355 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:28.072546 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:28.506930 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 81: this message will repeat every 5th retry.
W20251209 20:58:28.534327 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 81: this message will repeat every 5th retry.
W20251209 20:58:30.881950 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 81: this message will repeat every 5th retry.
I20251209 20:58:33.074030 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:33.074204 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:33.892530 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 81: this message will repeat every 5th retry.
I20251209 20:58:37.217942 15705 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:37.218097 15705 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:37.218706 15705 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:37.219547 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:58:37.220609 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:58:37.220754 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:58:37.220880 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:37.221097 15705 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:58:38.075587 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:38.075783 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:38.876654 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 86: this message will repeat every 5th retry.
W20251209 20:58:38.888612 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 86: this message will repeat every 5th retry.
I20251209 20:58:39.610200 15733 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:58:39.610356 15733 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:58:39.611104 15733 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:58:39.611950 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:58:39.612082 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:58:39.612257 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:58:39.612491 15733 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:58:40.768571 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 86: this message will repeat every 5th retry.
I20251209 20:58:43.076982 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:43.077171 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:44.695950 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 86: this message will repeat every 5th retry.
I20251209 20:58:48.078471 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:48.078640 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:58:48.507961 15851 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:48.508124 15851 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:48.508816 15851 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:48.509318 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:58:48.510638 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:58:48.510795 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:58:48.510944 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:48.511176 15851 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:58:48.907859 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 91: this message will repeat every 5th retry.
W20251209 20:58:49.481897 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 91: this message will repeat every 5th retry.
W20251209 20:58:50.821035 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 91: this message will repeat every 5th retry.
I20251209 20:58:53.079838 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:53.080011 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:58:53.663971 15892 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:58:53.664155 15892 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:58:53.664767 15892 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:58:53.665740 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:58:53.665920 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:58:53.666009 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:58:53.666241 15892 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:58:54.489157 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 91: this message will repeat every 5th retry.
I20251209 20:58:58.081393 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:58:58.081615 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:58:58.672493 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 96: this message will repeat every 5th retry.
I20251209 20:58:58.765281 15940 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:58:58.765463 15940 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:58:58.766177 15940 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:58:58.767043 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:58:58.768220 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:58:58.768400 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:58:58.768505 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:58:58.768721 15940 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:58:59.004912 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 96: this message will repeat every 5th retry.
W20251209 20:59:00.402838 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 96: this message will repeat every 5th retry.
I20251209 20:59:03.082861 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:03.083086 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:04.567132 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 96: this message will repeat every 5th retry.
I20251209 20:59:08.084537 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:08.084749 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:59:08.129031 16013 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:59:08.129202 16013 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:59:08.129859 16013 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:59:08.130335 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:59:08.131647 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:59:08.131873 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:59:08.131991 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:59:08.132324 16013 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:59:08.413311 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 101: this message will repeat every 5th retry.
W20251209 20:59:09.328182 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 101: this message will repeat every 5th retry.
W20251209 20:59:09.816075 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 101: this message will repeat every 5th retry.
I20251209 20:59:12.436748 16001 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:59:12.436976 16001 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:59:12.437808 16001 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:59:12.438619 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:59:12.438769 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:59:12.438913 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:59:12.439145 16001 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:59:13.086292 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:13.086496 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:15.048434 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 101: this message will repeat every 5th retry.
I20251209 20:59:18.087787 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:18.087975 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:18.160573 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 106: this message will repeat every 5th retry.
W20251209 20:59:19.849818 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 106: this message will repeat every 5th retry.
W20251209 20:59:20.204573 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 106: this message will repeat every 5th retry.
I20251209 20:59:23.089359 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:23.089578 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:23.805862 12583 heartbeater.cc:646] Failed to heartbeat to 127.11.132.254:40601 (953 consecutive failures): Network error: Failed to ping master at 127.11.132.254:40601: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:59:23.837661 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (426 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:59:24.136695 12717 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (426 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:59:25.203972 16148 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:59:25.204178 16148 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:59:25.204960 16148 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:59:25.205880 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:59:25.206887 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:59:25.207108 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:59:25.207216 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:59:25.207470 16148 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 20:59:25.441978 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 106: this message will repeat every 5th retry.
I20251209 20:59:26.485284 16140 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:59:26.485467 16140 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:59:26.486130 16140 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:59:26.486965 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:59:26.487092 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:59:26.487252 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:59:26.487514 16140 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:59:28.090865 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:28.091046 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:28.235455 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 111: this message will repeat every 5th retry.
W20251209 20:59:29.674789 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 111: this message will repeat every 5th retry.
W20251209 20:59:30.074302 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 111: this message will repeat every 5th retry.
I20251209 20:59:31.589215 16191 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:59:31.589385 16191 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:59:31.590119 16191 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:59:31.590764 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:59:31.590895 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:59:31.591034 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:59:31.591230 16191 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 20:59:33.092784 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:33.092978 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:36.073011 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 111: this message will repeat every 5th retry.
I20251209 20:59:38.094240 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:38.094427 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:38.743649 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 116: this message will repeat every 5th retry.
W20251209 20:59:39.411953 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 116: this message will repeat every 5th retry.
W20251209 20:59:40.474428 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 116: this message will repeat every 5th retry.
I20251209 20:59:43.096144 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:43.096355 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 20:59:43.624461 16339 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:59:43.624783 16339 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:59:43.625615 16339 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:59:43.626493 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 20:59:43.627548 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 20:59:43.627756 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 20:59:43.627882 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:59:43.628104 16339 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:59:45.295047 16354 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 20:59:45.295239 16354 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 20:59:45.296933 16354 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 20:59:45.297629 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 20:59:45.297809 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 20:59:45.297973 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 20:59:45.298211 16354 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 20:59:46.561610 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 116: this message will repeat every 5th retry.
I20251209 20:59:47.471199 16387 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 20:59:47.471390 16387 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 20:59:47.472003 16387 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 20:59:47.473626 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 20:59:47.473775 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 20:59:47.473901 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 20:59:47.474136 16387 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 20:59:48.098162 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:48.098371 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:48.487709 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 121: this message will repeat every 5th retry.
W20251209 20:59:49.306197 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 121: this message will repeat every 5th retry.
W20251209 20:59:50.546412 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 121: this message will repeat every 5th retry.
I20251209 20:59:53.099723 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:53.099925 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:56.623790 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 121: this message will repeat every 5th retry.
I20251209 20:59:58.101282 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 20:59:58.101459 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 20:59:58.338217 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 126: this message will repeat every 5th retry.
W20251209 20:59:59.309367 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 126: this message will repeat every 5th retry.
W20251209 21:00:00.195236 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 126: this message will repeat every 5th retry.
I20251209 21:00:03.103024 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:03.103201 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 21:00:03.786053 16591 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:03.786249 16591 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:03.787030 16591 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:03.788053 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251209 21:00:03.789191 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:03.789354 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:03.789489 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:03.789762 16591 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 21:00:05.162887 16560 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 21:00:05.163086 16560 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 21:00:05.164069 16560 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 21:00:05.164971 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 21:00:05.165124 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 21:00:05.165264 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 21:00:05.165486 16560 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 21:00:05.729542 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 126: this message will repeat every 5th retry.
W20251209 21:00:07.903466 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 131: this message will repeat every 5th retry.
I20251209 21:00:08.104588 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:08.104789 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:08.820741 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 131: this message will repeat every 5th retry.
W20251209 21:00:10.466854 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 131: this message will repeat every 5th retry.
I20251209 21:00:13.106447 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:13.106665 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 21:00:13.858789 16574 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 21:00:13.858995 16574 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 21:00:13.859689 16574 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 21:00:13.860597 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 21:00:13.860736 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 21:00:13.860841 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 21:00:13.861045 16574 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 21:00:15.388167 16701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:15.388360 16701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:15.389053 16701 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:15.392409 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:15.392409 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251209 21:00:15.393661 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:15.393831 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:15.393967 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:15.394218 16701 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 21:00:15.600438 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 131: this message will repeat every 5th retry.
W20251209 21:00:17.395776 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 136: this message will repeat every 5th retry.
I20251209 21:00:18.108064 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:18.108256 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:18.386703 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 136: this message will repeat every 5th retry.
W20251209 21:00:19.643587 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 136: this message will repeat every 5th retry.
I20251209 21:00:23.109509 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:23.109704 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:23.970827 12444 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (546 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 21:00:23.980556 12582 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (1251 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 21:00:24.305856 12717 heartbeater.cc:646] Failed to heartbeat to 127.11.132.252:42679 (546 consecutive failures): Network error: Failed to ping master at 127.11.132.252:42679: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:24.988713 16813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:24.988911 16813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:24.989547 16813 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:24.990458 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:24.991374 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
W20251209 21:00:24.991518 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
I20251209 21:00:24.991649 14141 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:24.991858 16813 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 21:00:26.657374 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 136: this message will repeat every 5th retry.
W20251209 21:00:27.502746 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 141: this message will repeat every 5th retry.
I20251209 21:00:28.111032 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:28.111267 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:28.609158 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 141: this message will repeat every 5th retry.
I20251209 21:00:29.713469 16880 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:29.713682 16880 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:29.714334 16880 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:29.716495 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:29.716688 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:29.716802 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:29.717005 16880 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 21:00:30.087033 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 141: this message will repeat every 5th retry.
I20251209 21:00:32.255895 16915 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 21:00:32.256047 16915 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 21:00:32.256799 16915 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 21:00:32.257488 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 21:00:32.257628 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 21:00:32.257762 12472 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 21:00:32.258018 16915 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251209 21:00:33.112815 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:33.113016 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 21:00:35.797753 16958 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:35.797960 16958 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:35.798669 16958 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:35.799474 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251209 21:00:35.800635 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:35.800782 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:35.800884 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:35.801093 16958 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251209 21:00:36.041829 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 141: this message will repeat every 5th retry.
W20251209 21:00:37.391974 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 146: this message will repeat every 5th retry.
W20251209 21:00:37.850997 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 146: this message will repeat every 5th retry.
I20251209 21:00:38.114390 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:38.114559 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:39.740756 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 146: this message will repeat every 5th retry.
I20251209 21:00:40.161074 17005 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ec484016839448df924d26e6175a3176)
I20251209 21:00:40.161243 17005 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec484016839448df924d26e6175a3176" member_type: VOTER last_known_addr { host: "127.11.132.254" port: 40601 } } peers { permanent_uuid: "1cc9184e29d6430ca18a3fe7984c725c" member_type: VOTER last_known_addr { host: "127.11.132.253" port: 34647 } } peers { permanent_uuid: "752a47ea6a344edba26b69df6ea3d482" member_type: VOTER last_known_addr { host: "127.11.132.252" port: 42679 } }
I20251209 21:00:40.161898 17005 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ec484016839448df924d26e6175a3176 (127.11.132.254:40601), 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679)
W20251209 21:00:40.163877 14141 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ec484016839448df924d26e6175a3176 (127.11.132.254:40601): Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:40.164252 14142 leader_election.cc:336] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 752a47ea6a344edba26b69df6ea3d482 (127.11.132.252:42679): Network error: Client connection negotiation failed: client connection to 127.11.132.252:42679: connect: Connection refused (error 111)
I20251209 21:00:40.164351 14142 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cc9184e29d6430ca18a3fe7984c725c; no voters: 752a47ea6a344edba26b69df6ea3d482, ec484016839448df924d26e6175a3176
I20251209 21:00:40.164593 17005 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1cc9184e29d6430ca18a3fe7984c725c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251209 21:00:43.115866 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:43.116031 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251209 21:00:44.839361 17049 raft_consensus.cc:493] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 157c709733f1484799756549aa456c5f)
I20251209 21:00:44.839540 17049 raft_consensus.cc:515] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cb2248952314571831b6958a0f0a7b4" member_type: VOTER last_known_addr { host: "127.11.132.194" port: 41287 } } peers { permanent_uuid: "157c709733f1484799756549aa456c5f" member_type: VOTER last_known_addr { host: "127.11.132.195" port: 38447 } } peers { permanent_uuid: "412a1acaab96447b97a475fb3f94fa12" member_type: VOTER last_known_addr { host: "127.11.132.193" port: 43197 } }
I20251209 21:00:44.840204 17049 leader_election.cc:290] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 157c709733f1484799756549aa456c5f (127.11.132.195:38447), 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197)
W20251209 21:00:44.841071 12472 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
W20251209 21:00:44.841195 12470 leader_election.cc:343] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Not found: Tablet not found: e31b692527c64e7389889b466bf0ebae
I20251209 21:00:44.841343 12470 leader_election.cc:304] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3cb2248952314571831b6958a0f0a7b4; no voters: 157c709733f1484799756549aa456c5f, 412a1acaab96447b97a475fb3f94fa12
I20251209 21:00:44.841559 17049 raft_consensus.cc:2749] T e31b692527c64e7389889b466bf0ebae P 3cb2248952314571831b6958a0f0a7b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251209 21:00:47.115916 12470 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 146: this message will repeat every 5th retry.
W20251209 21:00:47.829639 12472 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 151: this message will repeat every 5th retry.
W20251209 21:00:47.945618 12470 consensus_peers.cc:597] T 2f24e3afd3e24619a10ba98c02ae0899 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 412a1acaab96447b97a475fb3f94fa12 (127.11.132.193:43197): Couldn't send request to peer 412a1acaab96447b97a475fb3f94fa12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f24e3afd3e24619a10ba98c02ae0899. This is attempt 151: this message will repeat every 5th retry.
I20251209 21:00:48.117486 14126 master_runner.cc:186] Error getting master registration for 127.11.132.254:40601: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111),
I20251209 21:00:48.117689 14126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251209 21:00:49.317219 12472 consensus_peers.cc:597] T 34b3fe48f1d640a684f3aeb35586b354 P 3cb2248952314571831b6958a0f0a7b4 -> Peer 157c709733f1484799756549aa456c5f (127.11.132.195:38447): Couldn't send request to peer 157c709733f1484799756549aa456c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b3fe48f1d640a684f3aeb35586b354. This is attempt 151: this message will repeat every 5th retry.
W20251209 21:00:50.988266 12725 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:50.988505 12725 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:50.988605 12725 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
W20251209 21:00:50.988665 12725 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
F20251209 21:00:50.999384 12758 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1765314050 (unix time) try "date -d @1765314050" if you are using GNU date ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] C*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004087 12738 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_671e465611be47418f159b3c5ee7aa16 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004087 12738 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_671e465611be47418f159b3c5ee7aa16 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.005234 12753 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800002e13) received by PID 11795 (TID 0x7f495a9c1700) from PID 11795; stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004087 12738 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_671e465611be47418f159b3c5ee7aa16 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.005234 12753 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.006574 12745 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004087 12738 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_671e465611be47418f159b3c5ee7aa16 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.005234 12753 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.006574 12745 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.006611 12748 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f4979e88980 (unknown) at ??:0
@ 0x7f4972c5dfb7 gsignal at ??:0
F20251209 21:00:50.999507 12733 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a4904681b195452fb7adf16df235f461 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:50.999533 12734 master-stress-test.cc:297] Check failed: _s.ok() Bad status: F20251209 21:00:50.999533 12737 master-stress-test.cc:297] Timed out: Error creating table default.Table_afc836c7c2664ec0bd5f96ce0208fa81 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111) Timed out: Error creating table default.Table_cf01c2d93fcb46cba2bbea670526b180 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000780 12740 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3d73b33061d7464da6fc7c9367ad163b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.000977 12750 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001129 12749 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001308 12754 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001299 12735 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1b5967177d41405ababac4e174a63fc5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001489 12732 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_4a33bf9f531f4dc6bcfb90a1482cae4e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001566 12741 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_dcb97bd9ec28454eb4f5627100a4e5bb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.001302 12752 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002451 12757 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002691 12746 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002689 12744 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002847 12736 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b7b340bc7e5b46b39332ec8149739649 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.002822 12743 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.003947 12742 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004009 12747 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.004087 12738 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_671e465611be47418f159b3c5ee7aa16 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.005234 12753 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.006574 12745 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.006611 12748 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)F20251209 21:00:51.008788 12739 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a5e0fd74f125440aba983c391b8177ac on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.132.254:40601,127.11.132.253:34647,127.11.132.252:42679, num_attempts: 767) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.132.254:40601: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f4972c5f921 abort at ??:0
@ 0x7f4975995dcd google::LogMessage::Fail() at ??:0
@ 0x7f4975999b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f49759957cc google::LogMessage::Flush() at ??:0
@ 0x7f4975996f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40d9f5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f49732e56df (unknown) at ??:0
@ 0x7f4979e7d6db start_thread at ??:0
@ 0x7f4972d4071f clone at ??:0