Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250903 08:12:19.515906 13289 test_util.cc:276] Using random seed: 702315984
2025-09-03T08:12:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T08:12:19Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskVDqtkV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskVDqtkV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/security.properties
2025-09-03 08:12:25: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskVDqtkV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskVDqtkV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250903 08:12:33.398718 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=0
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:12:33.718948 13530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:33.719293 13530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:33.719412 13530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:33.728166 13530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:33.728271 13530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:33.728345 13530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:33.728385 13530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:33.738358 13530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:33.740101 13530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:33.742000 13530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:33.750767 13536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:33.750814 13535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:33.751377 13538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:33.751870 13530 server_base.cc:1047] running on GCE node
I20250903 08:12:33.752722 13530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:33.754238 13530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:33.755460 13530 hybrid_clock.cc:648] HybridClock initialized: now 1756887153755414 us; error 34 us; skew 500 ppm
I20250903 08:12:33.755836 13530 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:33.758147 13530 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:33.758916 13530 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:33.759024 13530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:33.759399 13530 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:33.761965 13530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/instance:
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:33.762753 13530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal/instance:
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:33.768859 13530 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250903 08:12:33.772541 13544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:33.774017 13530 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:12:33.774214 13530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:33.774422 13530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:33.800375 13530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:33.801841 13530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:33.802120 13530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:33.823822 13595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:33.823845 13530 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:33.825714 13530 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:33.829653 13596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:33.836560 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 13530
I20250903 08:12:33.836920 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal/instance
I20250903 08:12:33.836613 13596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:33.840305 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.125:34047
--webserver_interface=127.12.250.125
--webserver_port=0
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 08:12:33.855100 13596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
W20250903 08:12:33.856360 13546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:33.858014 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:33.907441 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } attempt: 1
W20250903 08:12:33.909572 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:33.977855 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } attempt: 2
W20250903 08:12:33.979772 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:34.071141 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } attempt: 3
W20250903 08:12:34.073197 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:34.193001 13599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:34.193390 13599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:34.193480 13599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:34.202257 13599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:34.202368 13599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:34.202427 13599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:34.202471 13599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:34.212495 13599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.125:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.250.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:34.214164 13599 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:34.215987 13599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 08:12:34.216507 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } attempt: 4
W20250903 08:12:34.218569 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:34.225140 13605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:34.225523 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:34.225968 13608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:34.226718 13599 server_base.cc:1047] running on GCE node
I20250903 08:12:34.227527 13599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:34.229115 13599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:34.230351 13599 hybrid_clock.cc:648] HybridClock initialized: now 1756887154230287 us; error 51 us; skew 500 ppm
I20250903 08:12:34.230737 13599 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:34.232997 13599 webserver.cc:480] Webserver started at http://127.12.250.125:36425/ using document root <none> and password file <none>
I20250903 08:12:34.233837 13599 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:34.233956 13599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:34.234308 13599 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:34.237300 13599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/instance:
uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.238155 13599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal/instance:
uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.243649 13599 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 08:12:34.247195 13614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:34.248600 13599 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:12:34.248781 13599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.248989 13599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:34.269518 13599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:34.270468 13599 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:34.270749 13599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:34.291709 13599 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.125:34047
I20250903 08:12:34.291741 13665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.125:34047 every 8 connection(s)
I20250903 08:12:34.293310 13599 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
I20250903 08:12:34.297703 13666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:34.301491 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 13599
I20250903 08:12:34.301671 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal/instance
I20250903 08:12:34.303002 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.124:39503
--webserver_interface=127.12.250.124
--webserver_port=0
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 08:12:34.304922 13666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:34.319742 13666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:34.329759 13666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
W20250903 08:12:34.331275 13615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:34.332845 13666 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.382134 13666 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 1
W20250903 08:12:34.383988 13666 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.452315 13666 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 2
W20250903 08:12:34.454306 13666 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.517884 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } attempt: 5
I20250903 08:12:34.524616 13596 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
W20250903 08:12:34.526862 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.545514 13666 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 3
W20250903 08:12:34.547398 13666 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.578131 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 1
W20250903 08:12:34.580271 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.648546 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 2
W20250903 08:12:34.650453 13596 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:34.653084 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:34.653477 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:34.653609 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:34.662281 13667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:34.662381 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:34.662436 13667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:34.662482 13667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:34.672060 13667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.124:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.250.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:34.673796 13667 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:34.675469 13667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:34.683424 13676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:34.683473 13675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:34.684167 13678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:34.685194 13667 server_base.cc:1047] running on GCE node
I20250903 08:12:34.685966 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:34.687351 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:34.688565 13667 hybrid_clock.cc:648] HybridClock initialized: now 1756887154688507 us; error 52 us; skew 500 ppm
I20250903 08:12:34.688874 13667 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:34.690626 13666 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 4
I20250903 08:12:34.691283 13667 webserver.cc:480] Webserver started at http://127.12.250.124:32903/ using document root <none> and password file <none>
I20250903 08:12:34.692234 13667 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:34.692353 13667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250903 08:12:34.692394 13666 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:34.692797 13667 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:34.695923 13667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/instance:
uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.696735 13667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal/instance:
uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.702389 13667 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250903 08:12:34.705878 13684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:34.707223 13667 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250903 08:12:34.707394 13667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:34.707613 13667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:34.726994 13667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:34.727941 13667 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:34.728665 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:34.749583 13667 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.124:39503
I20250903 08:12:34.749605 13735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.124:39503 every 8 connection(s)
I20250903 08:12:34.751127 13667 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
I20250903 08:12:34.752571 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 13667
I20250903 08:12:34.752786 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal/instance
I20250903 08:12:34.756726 13596 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 3
I20250903 08:12:34.756906 13736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:34.769927 13736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:34.784109 13596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:34.785197 13736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:34.788384 13596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:34.789959 13596 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:34.794554 13596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: No bootstrap required, opened a new log
I20250903 08:12:34.794632 13736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } has no permanent_uuid. Determining permanent_uuid...
I20250903 08:12:34.800268 13596 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:34.800678 13596 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:34.800807 13596 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:34.801579 13596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:34.803323 13744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:34.803700 13744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:34.803951 13596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
W20250903 08:12:34.804806 13550 tablet.cc:2378] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:12:34.806013 13736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Bootstrap starting.
I20250903 08:12:34.809970 13736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:34.812086 13736 log.cc:826] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:34.814997 13736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: No bootstrap required, opened a new log
I20250903 08:12:34.823733 13736 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:34.824862 13736 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:34.825052 13736 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9eecf53ccf4633bc701ff21e64ab0b, State: Initialized, Role: FOLLOWER
I20250903 08:12:34.819522 13530 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: OK, instance_id { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" instance_seqno: 1756887154285591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250903 08:12:34.826184 13736 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:34.828050 13530 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:34.843807 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:34.844269 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
W20250903 08:12:34.846347 13758 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:12:34.846618 13758 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:12:34.848421 13736 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:34.918094 13667 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: OK, instance_id { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" instance_seqno: 1756887154285591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250903 08:12:34.922398 13667 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:12:34.925552 13776 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:12:34.925805 13776 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250903 08:12:34.980980 13690 tablet.cc:2378] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:12:34.991631 13666 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } attempt: 5
I20250903 08:12:35.005519 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Bootstrap starting.
I20250903 08:12:35.010563 13666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:35.012081 13666 log.cc:826] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:35.015579 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: No bootstrap required, opened a new log
I20250903 08:12:35.024853 13666 raft_consensus.cc:357] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.025483 13666 raft_consensus.cc:383] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:35.025651 13666 raft_consensus.cc:738] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba49201780c41ebb58a4c8c0ae7bb1e, State: Initialized, Role: FOLLOWER
I20250903 08:12:35.026633 13666 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.027747 13780 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.028735 13780 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:35.029201 13666 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:35.062036 13744 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:35.062319 13744 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.064030 13744 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:35.065174 13641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" is_pre_election: true
I20250903 08:12:35.065208 13711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" is_pre_election: true
I20250903 08:12:35.065784 13641 raft_consensus.cc:2466] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 0.
I20250903 08:12:35.065811 13711 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 0.
I20250903 08:12:35.066619 13546 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters:
W20250903 08:12:35.075465 13795 catalog_manager.cc:1560] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 08:12:35.075671 13795 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 08:12:35.076596 13744 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:35.077286 13744 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:35.077720 13744 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:35.083104 13744 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.083675 13599 master_runner.cc:263] No leader master found from master dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:12:35.083858 13599 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:35.086380 13641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
I20250903 08:12:35.086601 13641 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:35.086936 13744 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 1 election: Requested vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:35.090067 13641 raft_consensus.cc:2466] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 1.
I20250903 08:12:35.091183 13711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
I20250903 08:12:35.091406 13711 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:35.095223 13546 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters:
I20250903 08:12:35.095613 13744 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:35.102138 13711 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 1.
I20250903 08:12:35.113065 13744 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 LEADER]: Becoming Leader. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Running, Role: LEADER
I20250903 08:12:35.114008 13744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.126798 13797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.127144 13797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:35.137300 13799 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:12:35.139802 13799 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:12:35.160161 13711 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Refusing update from remote peer 1e5331161d9e4df39004183d79dc3611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:12:35.161345 13797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:35.169117 13641 raft_consensus.cc:1273] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Refusing update from remote peer 1e5331161d9e4df39004183d79dc3611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:12:35.170658 13797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:35.207738 13780 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.207976 13780 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:35.211450 13800 mvcc.cc:204] Tried to move back new op lower bound from 7196209787525980160 to 7196209787353702400. Current Snapshot: MvccSnapshot[applied={T|T < 7196209787525980160}]
I20250903 08:12:35.216081 13802 mvcc.cc:204] Tried to move back new op lower bound from 7196209787525980160 to 7196209787353702400. Current Snapshot: MvccSnapshot[applied={T|T < 7196209787525980160}]
I20250903 08:12:35.217842 13780 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.218084 13780 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:35.219705 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.219964 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:35.224020 13801 mvcc.cc:204] Tried to move back new op lower bound from 7196209787525980160 to 7196209787353702400. Current Snapshot: MvccSnapshot[applied={T|T < 7196209787525980160}]
I20250903 08:12:35.225930 13744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.226272 13744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:35.226724 13744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.226928 13744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:35.231685 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:35.231943 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:35.233475 13799 catalog_manager.cc:1349] Generated new cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:35.233599 13799 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:12:35.302109 13799 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:12:35.308979 13799 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:12:35.333606 13799 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Generated new TSK 0
I20250903 08:12:35.334888 13799 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:12:35.336139 13799 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:12:35.343116 13799 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:12:35.379310 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.250.65:0
--local_ip_for_outbound_sockets=127.12.250.65
--webserver_interface=127.12.250.65
--webserver_port=0
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250903 08:12:35.765794 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:35.766108 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:35.766251 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:35.775316 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:35.775529 13808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.250.65
I20250903 08:12:35.785501 13808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.250.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.250.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.250.65
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:35.787192 13808 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:35.788995 13808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:35.802734 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:35.802676 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:35.803383 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:35.803875 13808 server_base.cc:1047] running on GCE node
I20250903 08:12:35.804507 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:35.805651 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:35.806890 13808 hybrid_clock.cc:648] HybridClock initialized: now 1756887155806814 us; error 63 us; skew 500 ppm
I20250903 08:12:35.807200 13808 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:35.809571 13808 webserver.cc:480] Webserver started at http://127.12.250.65:42075/ using document root <none> and password file <none>
I20250903 08:12:35.810349 13808 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:35.810461 13808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:35.810818 13808 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:35.813570 13808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data/instance:
uuid: "70c467196f6846b78b5a7c7ac8a2787d"
format_stamp: "Formatted at 2025-09-03 08:12:35 on dist-test-slave-knz7"
I20250903 08:12:35.814349 13808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/instance:
uuid: "70c467196f6846b78b5a7c7ac8a2787d"
format_stamp: "Formatted at 2025-09-03 08:12:35 on dist-test-slave-knz7"
I20250903 08:12:35.820700 13808 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250903 08:12:35.832190 13822 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:35.834545 13808 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.008s sys 0.000s
I20250903 08:12:35.834762 13808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal
uuid: "70c467196f6846b78b5a7c7ac8a2787d"
format_stamp: "Formatted at 2025-09-03 08:12:35 on dist-test-slave-knz7"
I20250903 08:12:35.835048 13808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:35.843961 13824 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:35.844285 13824 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:35.846086 13824 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:35.847054 13571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
I20250903 08:12:35.847395 13710 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" is_pre_election: true
I20250903 08:12:35.850478 13615 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e; no voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:35.851248 13824 raft_consensus.cc:2747] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 08:12:35.868139 13808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:35.869108 13808 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:35.869468 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:35.870489 13808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:12:35.872330 13808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:12:35.872447 13808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:35.872565 13808 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:12:35.872644 13808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:35.911063 13808 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.65:45323
I20250903 08:12:35.911108 13935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.65:45323 every 8 connection(s)
I20250903 08:12:35.912705 13808 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/data/info.pb
I20250903 08:12:35.913064 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 13808
I20250903 08:12:35.913292 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/instance
I20250903 08:12:35.919411 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.250.66:0
--local_ip_for_outbound_sockets=127.12.250.66
--webserver_interface=127.12.250.66
--webserver_port=0
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 08:12:35.928346 13776 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:35.928493 13776 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: loading cluster ID for follower catalog manager: success
I20250903 08:12:35.933348 13776 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: acquiring CA information for follower catalog manager: success
I20250903 08:12:35.936972 13776 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:35.942951 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:35.943475 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:35.944751 13937 heartbeater.cc:507] Master 127.12.250.126:42877 requested a full tablet report, sending...
I20250903 08:12:35.947455 13561 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:35.948215 13938 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:35.948366 13938 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:35.949450 13938 heartbeater.cc:507] Master 127.12.250.125:34047 requested a full tablet report, sending...
I20250903 08:12:35.949944 13561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.250.65:40269
I20250903 08:12:35.951265 13631 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:35.954865 13936 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:12:35.955004 13936 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:35.955324 13936 heartbeater.cc:507] Master 127.12.250.124:39503 requested a full tablet report, sending...
I20250903 08:12:35.956895 13701 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:36.077674 13795 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:36.077785 13795 catalog_manager.cc:1554] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: loading cluster ID for follower catalog manager: success
I20250903 08:12:36.081946 13795 catalog_manager.cc:1576] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: acquiring CA information for follower catalog manager: success
I20250903 08:12:36.084113 13795 catalog_manager.cc:1604] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250903 08:12:36.294997 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:36.295352 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:36.295498 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:36.304073 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:36.304298 13943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.250.66
I20250903 08:12:36.314296 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.250.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.250.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.250.66
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:36.315992 13943 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:36.317870 13943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:36.328480 13951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:36.328660 13952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:36.328997 13954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:36.329950 13943 server_base.cc:1047] running on GCE node
I20250903 08:12:36.330540 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:36.331452 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:36.332652 13943 hybrid_clock.cc:648] HybridClock initialized: now 1756887156332594 us; error 51 us; skew 500 ppm
I20250903 08:12:36.332963 13943 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:36.335089 13943 webserver.cc:480] Webserver started at http://127.12.250.66:40517/ using document root <none> and password file <none>
I20250903 08:12:36.335839 13943 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:36.335937 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:36.336303 13943 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:36.339275 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data/instance:
uuid: "668be13bcd854aa8afa97d2bbaf067a7"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.340077 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/instance:
uuid: "668be13bcd854aa8afa97d2bbaf067a7"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.346007 13943 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250903 08:12:36.349630 13960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.351163 13943 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:12:36.351374 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal
uuid: "668be13bcd854aa8afa97d2bbaf067a7"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.351608 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:36.385560 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:36.386502 13943 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:36.386792 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:36.387765 13943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:12:36.389505 13943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:12:36.389616 13943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.389747 13943 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:12:36.389824 13943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.424240 13943 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.66:42745
I20250903 08:12:36.424307 14072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.66:42745 every 8 connection(s)
I20250903 08:12:36.425843 13943 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/data/info.pb
I20250903 08:12:36.432240 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 13943
I20250903 08:12:36.432449 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/instance
I20250903 08:12:36.439322 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.250.67:0
--local_ip_for_outbound_sockets=127.12.250.67
--webserver_interface=127.12.250.67
--webserver_port=0
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 08:12:36.444006 14073 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:12:36.444479 14073 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.444501 14075 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:36.444698 14075 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.446156 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:36.446319 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.446329 14073 heartbeater.cc:507] Master 127.12.250.124:39503 requested a full tablet report, sending...
I20250903 08:12:36.446667 14075 heartbeater.cc:507] Master 127.12.250.125:34047 requested a full tablet report, sending...
I20250903 08:12:36.447000 14074 heartbeater.cc:507] Master 127.12.250.126:42877 requested a full tablet report, sending...
I20250903 08:12:36.448400 13631 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:36.448400 13701 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:36.449043 13561 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:36.449999 13561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.250.66:51123
I20250903 08:12:36.454141 13937 heartbeater.cc:499] Master 127.12.250.126:42877 was elected leader, sending a full tablet report...
W20250903 08:12:36.790581 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:36.790881 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:36.790977 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:36.799772 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:36.799953 14080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.250.67
I20250903 08:12:36.810081 14080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.250.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.250.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.250.67
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:36.811750 14080 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:36.813551 14080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:36.823424 14085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:36.823380 14088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:36.823467 14086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:36.827144 14080 server_base.cc:1047] running on GCE node
I20250903 08:12:36.827740 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:36.828785 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:36.830044 14080 hybrid_clock.cc:648] HybridClock initialized: now 1756887156829970 us; error 64 us; skew 500 ppm
I20250903 08:12:36.830406 14080 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:36.833150 14080 webserver.cc:480] Webserver started at http://127.12.250.67:40509/ using document root <none> and password file <none>
I20250903 08:12:36.834102 14080 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:36.834211 14080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:36.834625 14080 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:12:36.837661 14080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data/instance:
uuid: "0b68b07307444fdebbee0bff3fb79869"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.838392 14080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/instance:
uuid: "0b68b07307444fdebbee0bff3fb79869"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.844029 14080 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 08:12:36.847533 14094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.848937 14080 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250903 08:12:36.849095 14080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal
uuid: "0b68b07307444fdebbee0bff3fb79869"
format_stamp: "Formatted at 2025-09-03 08:12:36 on dist-test-slave-knz7"
I20250903 08:12:36.849360 14080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:36.868928 14080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:36.869896 14080 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:36.870187 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:36.871217 14080 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:12:36.872869 14080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:12:36.872951 14080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.873054 14080 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:12:36.873121 14080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:36.910470 14080 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.67:34257
I20250903 08:12:36.910565 14206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.67:34257 every 8 connection(s)
I20250903 08:12:36.912174 14080 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/data/info.pb
I20250903 08:12:36.922052 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14080
I20250903 08:12:36.922253 13289 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/instance
I20250903 08:12:36.931773 14209 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:36.932102 14207 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:12:36.932250 14209 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.932250 14207 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.932304 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:36.932782 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:36.933584 14207 heartbeater.cc:507] Master 127.12.250.124:39503 requested a full tablet report, sending...
I20250903 08:12:36.933640 14209 heartbeater.cc:507] Master 127.12.250.125:34047 requested a full tablet report, sending...
I20250903 08:12:36.933681 14208 heartbeater.cc:507] Master 127.12.250.126:42877 requested a full tablet report, sending...
I20250903 08:12:36.935436 13701 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:36.935420 13561 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:36.935595 13631 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:36.936491 13561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.250.67:46627
I20250903 08:12:36.951212 13289 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 08:12:36.952724 14074 heartbeater.cc:499] Master 127.12.250.126:42877 was elected leader, sending a full tablet report...
I20250903 08:12:36.982271 13558 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_3f66d15db9bc4be7bf75984378182586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.982955 13557 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_b21f9eea82374d75a52c4b6787afdf48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.983060 13556 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_517de34d608c42cc9d4aeafea81f283f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.983764 13560 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.984009 13555 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_fc56d0f82c054a9facf698e29514b8d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.986933 13554 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_feb51f00a18446ed842cbf54db2f3e06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:12:36.990316 13558 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3f66d15db9bc4be7bf75984378182586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:12:36.992726 13553 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:36.993937 13552 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.009600 13561 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_f7217c183b8a415d90880886722ccbb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.009632 13559 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.016377 13289 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu pid 13530
I20250903 08:12:37.017701 13591 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 08:12:37.173038 13289 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu with pid 13530
W20250903 08:12:37.203919 14214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.126:42877)
I20250903 08:12:37.205356 14254 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:37.205598 14254 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:37.207239 14254 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
I20250903 08:12:37.207942 13640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" is_pre_election: true
W20250903 08:12:37.210084 13685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:37.215045 13685 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
I20250903 08:12:37.215265 13685 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9eecf53ccf4633bc701ff21e64ab0b; no voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:12:37.216030 14254 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 08:12:37.281384 14257 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:37.281572 14257 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:37.282266 14257 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:37.283205 13710 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" is_pre_election: true
I20250903 08:12:37.283489 13710 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba49201780c41ebb58a4c8c0ae7bb1e in term 1.
I20250903 08:12:37.284400 13615 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b; no voters:
I20250903 08:12:37.284721 14257 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 08:12:37.284832 14257 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:37.284924 14257 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 1 FOLLOWER]: Advancing to term 2
W20250903 08:12:37.285218 13615 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
I20250903 08:12:37.287190 14257 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:37.287765 14257 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 election: Requested vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:37.288383 13710 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
I20250903 08:12:37.288630 13710 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 1 FOLLOWER]: Advancing to term 2
W20250903 08:12:37.289652 13615 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
I20250903 08:12:37.293059 13710 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba49201780c41ebb58a4c8c0ae7bb1e in term 2.
I20250903 08:12:37.293526 13615 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: 1e5331161d9e4df39004183d79dc3611
I20250903 08:12:37.293968 14257 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Leader election won for term 2
I20250903 08:12:37.294402 14257 raft_consensus.cc:695] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 LEADER]: Becoming Leader. State: Replica: dba49201780c41ebb58a4c8c0ae7bb1e, State: Running, Role: LEADER
I20250903 08:12:37.294934 14257 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:37.297015 14260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba49201780c41ebb58a4c8c0ae7bb1e. Latest consensus state: current_term: 2 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:37.297509 14260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:37.298246 14262 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:12:37.300170 14262 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:12:37.301003 14262 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:37.301077 14262 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:12:37.302151 14262 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:12:37.302891 14262 catalog_manager.cc:5966] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Loaded TSK: 0
I20250903 08:12:37.303755 14262 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:12:37.304502 14262 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:12:37.305295 14262 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:12:37.311968 13628 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_fc56d0f82c054a9facf698e29514b8d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.312346 13622 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_517de34d608c42cc9d4aeafea81f283f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.311730 13625 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.311303 13626 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_f7217c183b8a415d90880886722ccbb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.312619 13623 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_b21f9eea82374d75a52c4b6787afdf48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.311653 13627 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.312289 13629 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_feb51f00a18446ed842cbf54db2f3e06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.311995 13624 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_3f66d15db9bc4be7bf75984378182586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.310966 13630 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.310891 13631 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:12:37.330904 13625 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ab9942373e884c1490e9c1fb31e1b23b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:12:37.438706 14208 heartbeater.cc:499] Master 127.12.250.126:42877 was elected leader, sending a full tablet report...
W20250903 08:12:37.447623 14208 heartbeater.cc:646] Failed to heartbeat to 127.12.250.126:42877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:37.455895 14074 heartbeater.cc:646] Failed to heartbeat to 127.12.250.126:42877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:37.458706 13937 heartbeater.cc:646] Failed to heartbeat to 127.12.250.126:42877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
I20250903 08:12:37.501322 13625 catalog_manager.cc:2149] created HMS catalog entry for table [id=53236081245549cdbf54ad2a3c947bff]
W20250903 08:12:37.510753 13626 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f7217c183b8a415d90880886722ccbb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.510867 13631 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.511914 13622 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_517de34d608c42cc9d4aeafea81f283f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.512869 13623 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b21f9eea82374d75a52c4b6787afdf48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.513844 13627 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c04a578b55684bd69f07a226fc5dfdfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.514780 13629 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_feb51f00a18446ed842cbf54db2f3e06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.515659 13624 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3f66d15db9bc4be7bf75984378182586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.516512 13630 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cfed4bd13e9b4b9f843905f4874d6f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 08:12:37.517441 13628 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fc56d0f82c054a9facf698e29514b8d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:12:37.517802 13710 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Refusing update from remote peer dba49201780c41ebb58a4c8c0ae7bb1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250903 08:12:37.519281 14257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250903 08:12:37.521771 13615 consensus_peers.cc:489] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e -> Peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Couldn't send request to peer 1e5331161d9e4df39004183d79dc3611. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:12:37.536566 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba49201780c41ebb58a4c8c0ae7bb1e. Latest consensus state: current_term: 2 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:37.536813 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:37.540735 14257 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:37.541005 14257 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:37.543439 13631 catalog_manager.cc:2149] created HMS catalog entry for table [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:12:37.551000 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:37.551254 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:37.554523 13625 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b" } new_table_name: "default.Table_05bff585c5694de286caf65b794fa796" modify_external_catalogs: true
I20250903 08:12:37.560410 13631 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c" } new_table_name: "default.Table_21c2b2a8c5c14a4b86cfe245eb1e81e9" modify_external_catalogs: true
I20250903 08:12:37.594297 13626 catalog_manager.cc:2149] created HMS catalog entry for table [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:12:37.614408 14008 tablet_service.cc:1468] Processing CreateTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.614410 14142 tablet_service.cc:1468] Processing CreateTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.616114 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0302a6fea9242ef8efe0dc6a24d993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.615599 14141 tablet_service.cc:1468] Processing CreateTablet for tablet a43782057c894c77ad657f67deb11a71 (DEFAULT_TABLE table=default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.617544 13626 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_f7217c183b8a415d90880886722ccbb3" } new_table_name: "default.Table_3a84325d6c704721aaf5a796191718f2" modify_external_catalogs: true
I20250903 08:12:37.618028 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43782057c894c77ad657f67deb11a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.618005 14006 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca803c31f194e44b9b420030198fbff (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.618446 14006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca803c31f194e44b9b420030198fbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.616134 14140 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca803c31f194e44b9b420030198fbff (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.621776 14140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca803c31f194e44b9b420030198fbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.621743 14007 tablet_service.cc:1468] Processing CreateTablet for tablet a43782057c894c77ad657f67deb11a71 (DEFAULT_TABLE table=default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.622318 14007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43782057c894c77ad657f67deb11a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.622783 14142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0302a6fea9242ef8efe0dc6a24d993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.638587 14277 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.640806 14278 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.642760 14277 tablet_bootstrap.cc:654] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.643338 13871 tablet_service.cc:1468] Processing CreateTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.644384 14277 log.cc:826] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:37.644532 14278 tablet_bootstrap.cc:654] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.645015 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0302a6fea9242ef8efe0dc6a24d993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.646098 14278 log.cc:826] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:37.650096 13870 tablet_service.cc:1468] Processing CreateTablet for tablet a43782057c894c77ad657f67deb11a71 (DEFAULT_TABLE table=default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.650679 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43782057c894c77ad657f67deb11a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.653700 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca803c31f194e44b9b420030198fbff (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.654229 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca803c31f194e44b9b420030198fbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.670735 14281 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:37.681344 14278 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.681771 14278 ts_tablet_manager.cc:1397] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.026s
I20250903 08:12:37.690114 14281 tablet_bootstrap.cc:654] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.691596 14281 log.cc:826] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:37.691524 14278 raft_consensus.cc:357] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.692162 14278 raft_consensus.cc:383] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.692312 14278 raft_consensus.cc:738] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.693336 14278 consensus_queue.cc:260] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.698328 13622 catalog_manager.cc:2149] created HMS catalog entry for table [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:12:37.699258 14278 ts_tablet_manager.cc:1428] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20250903 08:12:37.699762 14209 heartbeater.cc:499] Master 127.12.250.125:34047 was elected leader, sending a full tablet report...
I20250903 08:12:37.701681 14277 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:37.702082 14277 ts_tablet_manager.cc:1397] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.046s
I20250903 08:12:37.709033 14281 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:37.709399 14281 ts_tablet_manager.cc:1397] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20250903 08:12:37.708971 14277 raft_consensus.cc:357] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.712986 14277 raft_consensus.cc:383] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.713194 14277 raft_consensus.cc:738] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.714166 14277 consensus_queue.cc:260] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.718012 14281 raft_consensus.cc:357] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.718428 14281 raft_consensus.cc:383] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.718566 14281 raft_consensus.cc:738] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.722441 13869 tablet_service.cc:1468] Processing CreateTablet for tablet f65826010a9b482faf67a7120247eab1 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.722472 14006 tablet_service.cc:1468] Processing CreateTablet for tablet f65826010a9b482faf67a7120247eab1 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.722788 14277 ts_tablet_manager.cc:1428] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.021s user 0.002s sys 0.014s
I20250903 08:12:37.722864 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65826010a9b482faf67a7120247eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.722896 14006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65826010a9b482faf67a7120247eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.728256 14075 heartbeater.cc:499] Master 127.12.250.125:34047 was elected leader, sending a full tablet report...
I20250903 08:12:37.729111 14277 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.729984 14141 tablet_service.cc:1468] Processing CreateTablet for tablet f65826010a9b482faf67a7120247eab1 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.730470 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65826010a9b482faf67a7120247eab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.731163 14277 tablet_bootstrap.cc:654] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.730132 14281 consensus_queue.cc:260] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.738754 13622 master_service.cc:948] ReplaceTablet: received request to replace tablet f0302a6fea9242ef8efe0dc6a24d993e from {username='slave'} at 127.0.0.1:34214
I20250903 08:12:37.740397 14277 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:37.740693 14277 ts_tablet_manager.cc:1397] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20250903 08:12:37.741024 14278 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.741916 14277 raft_consensus.cc:357] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.742293 14277 raft_consensus.cc:383] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.742417 14277 raft_consensus.cc:738] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.742785 14277 consensus_queue.cc:260] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.743126 13938 heartbeater.cc:499] Master 127.12.250.125:34047 was elected leader, sending a full tablet report...
I20250903 08:12:37.743475 14277 ts_tablet_manager.cc:1428] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:12:37.743841 14277 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.746405 14277 tablet_bootstrap.cc:654] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.748085 14281 ts_tablet_manager.cc:1428] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.039s user 0.013s sys 0.003s
I20250903 08:12:37.748724 14281 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:37.752154 14281 tablet_bootstrap.cc:654] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.758824 14277 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:37.758992 14277 ts_tablet_manager.cc:1397] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20250903 08:12:37.759826 14277 raft_consensus.cc:357] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.760020 14277 raft_consensus.cc:383] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.760114 14277 raft_consensus.cc:738] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.760385 14277 consensus_queue.cc:260] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.762822 14277 ts_tablet_manager.cc:1428] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250903 08:12:37.765668 14277 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.767319 14277 tablet_bootstrap.cc:654] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.769474 14281 tablet_bootstrap.cc:492] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:37.769670 14281 ts_tablet_manager.cc:1397] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.003s
I20250903 08:12:37.769994 13622 catalog_manager.cc:5869] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Sending DeleteTablet for 3 replicas of tablet f0302a6fea9242ef8efe0dc6a24d993e
I20250903 08:12:37.770750 14281 raft_consensus.cc:357] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.770973 14281 raft_consensus.cc:383] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.771059 14281 raft_consensus.cc:738] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.771412 14281 consensus_queue.cc:260] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.771615 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e with delete_type TABLET_DATA_DELETED (replaced by tablet 09070dc67d994d26a5c768db60faf96c) from {username='slave'} at 127.0.0.1:49834
I20250903 08:12:37.771960 14281 ts_tablet_manager.cc:1428] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:37.778029 14281 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:37.779112 14278 tablet_bootstrap.cc:654] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.779754 14281 tablet_bootstrap.cc:654] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.782645 14141 tablet_service.cc:1515] Processing DeleteTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e with delete_type TABLET_DATA_DELETED (replaced by tablet 09070dc67d994d26a5c768db60faf96c) from {username='slave'} at 127.0.0.1:56754
I20250903 08:12:37.783681 14006 tablet_service.cc:1515] Processing DeleteTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e with delete_type TABLET_DATA_DELETED (replaced by tablet 09070dc67d994d26a5c768db60faf96c) from {username='slave'} at 127.0.0.1:35206
W20250903 08:12:37.799197 13616 catalog_manager.cc:4908] TS 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257): delete failed for tablet f0302a6fea9242ef8efe0dc6a24d993e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0302a6fea9242ef8efe0dc6a24d993e already in progress: creating tablet
I20250903 08:12:37.800752 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 09070dc67d994d26a5c768db60faf96c (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.801180 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09070dc67d994d26a5c768db60faf96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.801455 14281 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:37.801419 14008 tablet_service.cc:1468] Processing CreateTablet for tablet 09070dc67d994d26a5c768db60faf96c (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.801602 14281 ts_tablet_manager.cc:1397] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20250903 08:12:37.801862 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09070dc67d994d26a5c768db60faf96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.802042 14141 tablet_service.cc:1468] Processing CreateTablet for tablet 09070dc67d994d26a5c768db60faf96c (DEFAULT_TABLE table=default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.802400 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09070dc67d994d26a5c768db60faf96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.806140 13622 catalog_manager.cc:6652] ReplaceTablet: tablet f0302a6fea9242ef8efe0dc6a24d993e deleted and replaced by tablet 09070dc67d994d26a5c768db60faf96c
I20250903 08:12:37.808928 14281 raft_consensus.cc:357] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.809252 14281 raft_consensus.cc:383] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.809422 14281 raft_consensus.cc:738] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.809710 14281 consensus_queue.cc:260] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.810876 14294 tablet_replica.cc:331] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: stopping tablet replica
I20250903 08:12:37.811645 14294 raft_consensus.cc:2241] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.812290 14294 raft_consensus.cc:2270] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.813354 14281 ts_tablet_manager.cc:1428] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
W20250903 08:12:37.815145 13618 catalog_manager.cc:4908] TS 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323): delete failed for tablet f0302a6fea9242ef8efe0dc6a24d993e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0302a6fea9242ef8efe0dc6a24d993e already in progress: creating tablet
I20250903 08:12:37.816959 14278 tablet_bootstrap.cc:492] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.817281 14278 ts_tablet_manager.cc:1397] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20250903 08:12:37.818007 14281 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:37.818697 14278 raft_consensus.cc:357] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.819926 14281 tablet_bootstrap.cc:654] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.822607 14277 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:37.822889 14277 ts_tablet_manager.cc:1397] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.039s
I20250903 08:12:37.823827 14277 raft_consensus.cc:357] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.824100 14277 raft_consensus.cc:383] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.824227 14277 raft_consensus.cc:738] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.824568 14277 consensus_queue.cc:260] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.825150 14277 ts_tablet_manager.cc:1428] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 08:12:37.825572 14277 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.827363 14277 tablet_bootstrap.cc:654] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.828783 14281 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:37.829013 14281 ts_tablet_manager.cc:1397] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250903 08:12:37.829851 14281 raft_consensus.cc:357] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.830113 14281 raft_consensus.cc:383] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.830238 14281 raft_consensus.cc:738] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.830652 14281 consensus_queue.cc:260] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.832783 14278 raft_consensus.cc:383] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.833631 14294 ts_tablet_manager.cc:1905] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.834357 14281 ts_tablet_manager.cc:1428] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250903 08:12:37.836357 14278 raft_consensus.cc:738] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.836678 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e with delete_type TABLET_DATA_DELETED (replaced by tablet 09070dc67d994d26a5c768db60faf96c) from {username='slave'} at 127.0.0.1:49834
I20250903 08:12:37.837109 14290 tablet_replica.cc:331] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: stopping tablet replica
I20250903 08:12:37.837373 14281 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:37.837759 14290 raft_consensus.cc:2241] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.838277 14290 raft_consensus.cc:2270] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.839351 14281 tablet_bootstrap.cc:654] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.840443 14290 ts_tablet_manager.cc:1905] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.847398 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 08:12:37.848140 14278 consensus_queue.cc:260] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.853668 14278 ts_tablet_manager.cc:1428] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20250903 08:12:37.854106 14278 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.856117 14278 tablet_bootstrap.cc:654] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.857393 14281 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:37.857594 14281 ts_tablet_manager.cc:1397] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20250903 08:12:37.858485 14281 raft_consensus.cc:357] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.858681 14281 raft_consensus.cc:383] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.858786 14281 raft_consensus.cc:738] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.859009 14281 consensus_queue.cc:260] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.860512 14281 ts_tablet_manager.cc:1428] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 08:12:37.861733 13623 catalog_manager.cc:2149] created HMS catalog entry for table [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:12:37.862632 14290 ts_tablet_manager.cc:1918] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.862800 14290 log.cc:1199] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/wals/f0302a6fea9242ef8efe0dc6a24d993e
I20250903 08:12:37.863265 14290 ts_tablet_manager.cc:1939] T f0302a6fea9242ef8efe0dc6a24d993e P 70c467196f6846b78b5a7c7ac8a2787d: Deleting consensus metadata
I20250903 08:12:37.866756 13618 catalog_manager.cc:4928] TS 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323): tablet f0302a6fea9242ef8efe0dc6a24d993e (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]) successfully deleted
I20250903 08:12:37.870505 13622 master_service.cc:948] ReplaceTablet: received request to replace tablet f0302a6fea9242ef8efe0dc6a24d993e from {username='slave'} at 127.0.0.1:34214
I20250903 08:12:37.871023 13622 master_service.cc:948] ReplaceTablet: received request to replace tablet f0302a6fea9242ef8efe0dc6a24d993e from {username='slave'} at 127.0.0.1:34214
I20250903 08:12:37.871518 13623 master_service.cc:948] ReplaceTablet: received request to replace tablet 5ca803c31f194e44b9b420030198fbff from {username='slave'} at 127.0.0.1:34214
I20250903 08:12:37.880213 14294 ts_tablet_manager.cc:1918] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.880420 14294 log.cc:1199] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/wals/f0302a6fea9242ef8efe0dc6a24d993e
I20250903 08:12:37.880967 14294 ts_tablet_manager.cc:1939] T f0302a6fea9242ef8efe0dc6a24d993e P 668be13bcd854aa8afa97d2bbaf067a7: Deleting consensus metadata
I20250903 08:12:37.883970 14278 tablet_bootstrap.cc:492] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.884133 14278 ts_tablet_manager.cc:1397] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250903 08:12:37.884953 14278 raft_consensus.cc:357] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.885143 14278 raft_consensus.cc:383] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.885229 14278 raft_consensus.cc:738] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.885501 14278 consensus_queue.cc:260] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } }
I20250903 08:12:37.886026 14278 ts_tablet_manager.cc:1428] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 08:12:37.886312 14278 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.888123 14278 tablet_bootstrap.cc:654] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.890234 14277 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:37.890411 14277 ts_tablet_manager.cc:1397] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.065s user 0.030s sys 0.029s
I20250903 08:12:37.891291 14277 raft_consensus.cc:357] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.891470 14277 raft_consensus.cc:383] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.891558 14277 raft_consensus.cc:738] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.893374 13623 catalog_manager.cc:5869] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Sending DeleteTablet for 3 replicas of tablet 5ca803c31f194e44b9b420030198fbff
I20250903 08:12:37.893846 13623 catalog_manager.cc:6652] ReplaceTablet: tablet 5ca803c31f194e44b9b420030198fbff deleted and replaced by tablet 73649395ff784d5cb8ffff12c3e45dbe
I20250903 08:12:37.894881 14141 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca803c31f194e44b9b420030198fbff with delete_type TABLET_DATA_DELETED (replaced by tablet 73649395ff784d5cb8ffff12c3e45dbe) from {username='slave'} at 127.0.0.1:56754
I20250903 08:12:37.895180 14293 tablet_replica.cc:331] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: stopping tablet replica
I20250903 08:12:37.895251 13623 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_517de34d608c42cc9d4aeafea81f283f" } new_table_name: "default.Table_97201f0afe374761ad6c85589bf4ff69" modify_external_catalogs: true
I20250903 08:12:37.895676 14293 raft_consensus.cc:2241] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.896103 14293 raft_consensus.cc:2270] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.897943 14293 ts_tablet_manager.cc:1905] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.898366 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca803c31f194e44b9b420030198fbff with delete_type TABLET_DATA_DELETED (replaced by tablet 73649395ff784d5cb8ffff12c3e45dbe) from {username='slave'} at 127.0.0.1:49834
I20250903 08:12:37.898644 14006 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca803c31f194e44b9b420030198fbff with delete_type TABLET_DATA_DELETED (replaced by tablet 73649395ff784d5cb8ffff12c3e45dbe) from {username='slave'} at 127.0.0.1:35206
I20250903 08:12:37.898733 14277 consensus_queue.cc:260] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.899154 13617 catalog_manager.cc:4928] TS 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745): tablet f0302a6fea9242ef8efe0dc6a24d993e (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]) successfully deleted
I20250903 08:12:37.900022 14290 tablet_replica.cc:331] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: stopping tablet replica
I20250903 08:12:37.900264 14290 raft_consensus.cc:2241] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.900876 14290 raft_consensus.cc:2270] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.901763 14277 ts_tablet_manager.cc:1428] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250903 08:12:37.871524 13622 master_service.cc:948] ReplaceTablet: received request to replace tablet 5ca803c31f194e44b9b420030198fbff from {username='slave'} at 127.0.0.1:34214
I20250903 08:12:37.902940 14290 ts_tablet_manager.cc:1905] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.903100 14141 tablet_service.cc:1515] Processing DeleteTablet for tablet f0302a6fea9242ef8efe0dc6a24d993e with delete_type TABLET_DATA_DELETED (replaced by tablet 09070dc67d994d26a5c768db60faf96c) from {username='slave'} at 127.0.0.1:56754
I20250903 08:12:37.903396 14278 tablet_bootstrap.cc:492] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.903554 14278 ts_tablet_manager.cc:1397] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250903 08:12:37.904361 14278 raft_consensus.cc:357] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.904556 14278 raft_consensus.cc:383] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.904624 14278 raft_consensus.cc:738] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.904844 14278 consensus_queue.cc:260] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.905414 14278 ts_tablet_manager.cc:1428] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:37.905674 14278 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.907270 14278 tablet_bootstrap.cc:654] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.909451 14294 tablet_replica.cc:331] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: stopping tablet replica
I20250903 08:12:37.909554 14290 ts_tablet_manager.cc:1918] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.909655 14294 raft_consensus.cc:2241] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.909696 14290 log.cc:1199] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/wals/5ca803c31f194e44b9b420030198fbff
I20250903 08:12:37.909844 14294 raft_consensus.cc:2270] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.921514 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 4a67d22967884a02a6a89937065d2569 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 08:12:37.921940 14212 tablet.cc:2378] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:12:37.922060 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a67d22967884a02a6a89937065d2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.925387 14290 ts_tablet_manager.cc:1939] T 5ca803c31f194e44b9b420030198fbff P 70c467196f6846b78b5a7c7ac8a2787d: Deleting consensus metadata
I20250903 08:12:37.926396 14006 tablet_service.cc:1468] Processing CreateTablet for tablet 4a67d22967884a02a6a89937065d2569 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.926836 14006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a67d22967884a02a6a89937065d2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.928059 14293 ts_tablet_manager.cc:1918] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.928208 14293 log.cc:1199] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/wals/5ca803c31f194e44b9b420030198fbff
I20250903 08:12:37.928606 14293 ts_tablet_manager.cc:1939] T 5ca803c31f194e44b9b420030198fbff P 0b68b07307444fdebbee0bff3fb79869: Deleting consensus metadata
W20250903 08:12:37.929481 14076 tablet.cc:2378] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:12:37.929702 13618 catalog_manager.cc:4928] TS 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323): tablet 5ca803c31f194e44b9b420030198fbff (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]) successfully deleted
I20250903 08:12:37.931053 14278 tablet_bootstrap.cc:492] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.931206 14278 ts_tablet_manager.cc:1397] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250903 08:12:37.932018 14278 raft_consensus.cc:357] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.932180 14278 raft_consensus.cc:383] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.932255 14278 raft_consensus.cc:738] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.932464 14278 consensus_queue.cc:260] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.932971 14278 ts_tablet_manager.cc:1428] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 08:12:37.933493 14141 tablet_service.cc:1468] Processing CreateTablet for tablet 4a67d22967884a02a6a89937065d2569 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.934038 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a67d22967884a02a6a89937065d2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.940047 14278 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.941782 14278 tablet_bootstrap.cc:654] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
W20250903 08:12:37.942585 13944 tablet.cc:2378] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:12:37.946306 14294 ts_tablet_manager.cc:1905] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.952214 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.952553 14008 tablet_service.cc:1468] Processing CreateTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.952642 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73649395ff784d5cb8ffff12c3e45dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.956771 13627 catalog_manager.cc:2149] created HMS catalog entry for table [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:12:37.959071 14141 tablet_service.cc:1468] Processing CreateTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe (DEFAULT_TABLE table=default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:37.960101 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73649395ff784d5cb8ffff12c3e45dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.965874 13627 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_94faf7a1e44247f6aeaed3759e0b4f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.968453 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73649395ff784d5cb8ffff12c3e45dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:37.975083 14294 ts_tablet_manager.cc:1918] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.975551 14277 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:37.975548 14294 log.cc:1199] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/wals/5ca803c31f194e44b9b420030198fbff
I20250903 08:12:37.976293 14294 ts_tablet_manager.cc:1939] T 5ca803c31f194e44b9b420030198fbff P 668be13bcd854aa8afa97d2bbaf067a7: Deleting consensus metadata
I20250903 08:12:37.977368 14277 tablet_bootstrap.cc:654] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.978518 13617 catalog_manager.cc:4928] TS 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745): tablet 5ca803c31f194e44b9b420030198fbff (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]) successfully deleted
I20250903 08:12:37.979357 14278 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:37.979581 14278 ts_tablet_manager.cc:1397] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.009s
I20250903 08:12:37.980333 14278 raft_consensus.cc:357] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.980607 14278 raft_consensus.cc:383] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:37.980705 14278 raft_consensus.cc:738] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:37.981029 14278 consensus_queue.cc:260] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.982299 14278 ts_tablet_manager.cc:1428] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 08:12:37.982630 14278 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:37.984163 14278 tablet_bootstrap.cc:654] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:37.984193 13622 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_0ecab4f09e824890b839f9ec45bfa3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:37.985554 14293 tablet_replica.cc:331] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: stopping tablet replica
I20250903 08:12:37.985770 14293 raft_consensus.cc:2241] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:37.985960 14293 raft_consensus.cc:2270] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:37.986719 14293 ts_tablet_manager.cc:1905] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:37.987294 13616 catalog_manager.cc:4928] TS 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257): tablet 5ca803c31f194e44b9b420030198fbff (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]) successfully deleted
I20250903 08:12:37.992964 14293 ts_tablet_manager.cc:1918] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 08:12:37.993561 14293 log.cc:1199] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/wals/f0302a6fea9242ef8efe0dc6a24d993e
I20250903 08:12:37.993811 14283 raft_consensus.cc:491] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:37.994230 14293 ts_tablet_manager.cc:1939] T f0302a6fea9242ef8efe0dc6a24d993e P 0b68b07307444fdebbee0bff3fb79869: Deleting consensus metadata
I20250903 08:12:37.994055 14283 raft_consensus.cc:513] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:37.996428 14283 leader_election.cc:290] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.009744 14007 tablet_service.cc:1468] Processing CreateTablet for tablet da45d19c545a45c2abff9c05d8c0b293 (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.009958 13616 catalog_manager.cc:4928] TS 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257): tablet f0302a6fea9242ef8efe0dc6a24d993e (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]) successfully deleted
I20250903 08:12:38.010259 14007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da45d19c545a45c2abff9c05d8c0b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.011541 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09070dc67d994d26a5c768db60faf96c" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.011950 14281 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.012193 14028 raft_consensus.cc:2466] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.013130 14097 leader_election.cc:304] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.013849 14281 tablet_bootstrap.cc:654] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.014169 13871 tablet_service.cc:1468] Processing CreateTablet for tablet da45d19c545a45c2abff9c05d8c0b293 (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.014720 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da45d19c545a45c2abff9c05d8c0b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.017575 14283 raft_consensus.cc:2802] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.017769 14283 raft_consensus.cc:491] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.017867 14283 raft_consensus.cc:3058] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.023984 14141 tablet_service.cc:1468] Processing CreateTablet for tablet da45d19c545a45c2abff9c05d8c0b293 (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.024480 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da45d19c545a45c2abff9c05d8c0b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.021169 14283 raft_consensus.cc:513] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.028026 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09070dc67d994d26a5c768db60faf96c" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.028308 14028 raft_consensus.cc:3058] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.032447 14028 raft_consensus.cc:2466] T 09070dc67d994d26a5c768db60faf96c P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.033072 14097 leader_election.cc:304] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.036218 14283 leader_election.cc:290] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.036577 14283 raft_consensus.cc:2802] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.036746 14283 raft_consensus.cc:695] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [term 1 LEADER]: Becoming Leader. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Running, Role: LEADER
I20250903 08:12:38.037326 14283 consensus_queue.cc:237] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.043735 14281 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.043951 14281 ts_tablet_manager.cc:1397] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.004s
I20250903 08:12:38.044799 14281 raft_consensus.cc:357] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.045104 14281 raft_consensus.cc:383] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.045199 14281 raft_consensus.cc:738] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.045588 14281 consensus_queue.cc:260] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.046132 14281 ts_tablet_manager.cc:1428] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:38.046293 14283 raft_consensus.cc:491] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.048187 14281 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.050544 14281 tablet_bootstrap.cc:654] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.056869 14277 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.057199 14277 ts_tablet_manager.cc:1397] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.022s
I20250903 08:12:38.057364 14283 raft_consensus.cc:513] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.058272 13891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09070dc67d994d26a5c768db60faf96c" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.058532 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.058900 13891 raft_consensus.cc:2466] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
W20250903 08:12:38.059525 14097 leader_election.cc:343] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745): Illegal state: must be running to vote when last-logged opid is not known
I20250903 08:12:38.059507 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.059832 13889 raft_consensus.cc:2466] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.060389 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09070dc67d994d26a5c768db60faf96c" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.060678 13890 raft_consensus.cc:3058] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.060853 14278 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:38.061003 14278 ts_tablet_manager.cc:1397] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.012s
I20250903 08:12:38.061931 14278 raft_consensus.cc:357] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.062109 14278 raft_consensus.cc:383] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.062193 14278 raft_consensus.cc:738] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.062441 14278 consensus_queue.cc:260] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.062971 14278 ts_tablet_manager.cc:1428] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:38.065340 13890 raft_consensus.cc:2466] T 09070dc67d994d26a5c768db60faf96c P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.066252 14098 leader_election.cc:304] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters: 668be13bcd854aa8afa97d2bbaf067a7
I20250903 08:12:38.066759 14278 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:38.067759 14309 raft_consensus.cc:2802] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.067942 14309 raft_consensus.cc:491] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.068116 14309 raft_consensus.cc:3058] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.068501 14277 raft_consensus.cc:357] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.068783 14277 raft_consensus.cc:383] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.068905 14277 raft_consensus.cc:738] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.069293 14277 consensus_queue.cc:260] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.069392 14283 leader_election.cc:290] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.070538 14277 ts_tablet_manager.cc:1428] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 08:12:38.070979 14277 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:38.072837 14277 tablet_bootstrap.cc:654] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.079941 13629 catalog_manager.cc:2149] created HMS catalog entry for table [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:12:38.081048 14309 raft_consensus.cc:513] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.084162 14309 leader_election.cc:290] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.084496 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.084630 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.084723 13890 raft_consensus.cc:3058] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.085040 14278 tablet_bootstrap.cc:654] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
W20250903 08:12:38.085202 14097 leader_election.cc:343] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745): Illegal state: must be running to vote when last-logged opid is not known
I20250903 08:12:38.087042 14281 tablet_bootstrap.cc:492] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.087260 14281 ts_tablet_manager.cc:1397] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.001s
I20250903 08:12:38.087841 13890 raft_consensus.cc:2466] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.088219 14281 raft_consensus.cc:357] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.088346 14098 leader_election.cc:304] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters: 668be13bcd854aa8afa97d2bbaf067a7
I20250903 08:12:38.088469 14281 raft_consensus.cc:383] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.088570 14281 raft_consensus.cc:738] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.088794 14309 raft_consensus.cc:2802] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.088999 14309 raft_consensus.cc:695] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [term 1 LEADER]: Becoming Leader. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Running, Role: LEADER
I20250903 08:12:38.089241 14281 consensus_queue.cc:260] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.089780 14277 tablet_bootstrap.cc:492] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.089977 14277 ts_tablet_manager.cc:1397] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20250903 08:12:38.091135 14277 raft_consensus.cc:357] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.091387 14277 raft_consensus.cc:383] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.091536 14277 raft_consensus.cc:738] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.091816 14277 consensus_queue.cc:260] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.092525 14277 ts_tablet_manager.cc:1428] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:38.092911 14277 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:38.094846 14277 tablet_bootstrap.cc:654] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.104462 14008 tablet_service.cc:1468] Processing CreateTablet for tablet bdfe060848fc4e9d9534e539f39137da (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.104902 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfe060848fc4e9d9534e539f39137da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.103672 14141 tablet_service.cc:1468] Processing CreateTablet for tablet bdfe060848fc4e9d9534e539f39137da (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.106814 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfe060848fc4e9d9534e539f39137da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.103627 13871 tablet_service.cc:1468] Processing CreateTablet for tablet bdfe060848fc4e9d9534e539f39137da (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.109787 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfe060848fc4e9d9534e539f39137da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.089742 14309 consensus_queue.cc:237] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.116753 14281 ts_tablet_manager.cc:1428] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250903 08:12:38.117093 14281 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.119022 14281 tablet_bootstrap.cc:654] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.121821 13629 catalog_manager.cc:5582] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67). New cstate: current_term: 1 leader_uuid: "0b68b07307444fdebbee0bff3fb79869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: HEALTHY } } }
I20250903 08:12:38.130084 14281 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.130486 14278 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:38.130666 14278 ts_tablet_manager.cc:1397] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.002s
I20250903 08:12:38.130666 14281 ts_tablet_manager.cc:1397] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250903 08:12:38.131606 14278 raft_consensus.cc:357] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.131836 14278 raft_consensus.cc:383] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.131920 14278 raft_consensus.cc:738] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.132203 14278 consensus_queue.cc:260] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.132725 14278 ts_tablet_manager.cc:1428] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 08:12:38.135257 14281 raft_consensus.cc:357] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.135581 14281 raft_consensus.cc:383] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.135723 14281 raft_consensus.cc:738] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.136070 14281 consensus_queue.cc:260] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.136392 13629 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_250ab81379d64db694b3b9013da7f8a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:38.136992 14278 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:38.138904 14278 tablet_bootstrap.cc:654] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.140376 14281 ts_tablet_manager.cc:1428] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250903 08:12:38.140550 14277 tablet_bootstrap.cc:492] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.140722 14277 ts_tablet_manager.cc:1397] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.005s
I20250903 08:12:38.141607 14277 raft_consensus.cc:357] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.141822 14277 raft_consensus.cc:383] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.141896 14277 raft_consensus.cc:738] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.142117 14277 consensus_queue.cc:260] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.142598 14277 ts_tablet_manager.cc:1428] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:12:38.142963 14277 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:38.144681 14277 tablet_bootstrap.cc:654] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.145213 14281 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.147092 14281 tablet_bootstrap.cc:654] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.154502 14277 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.154718 14277 ts_tablet_manager.cc:1397] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250903 08:12:38.155897 14277 raft_consensus.cc:357] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.156105 14277 raft_consensus.cc:383] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.156212 14277 raft_consensus.cc:738] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.156500 14277 consensus_queue.cc:260] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.156759 14281 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.156956 14281 ts_tablet_manager.cc:1397] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250903 08:12:38.157840 14281 raft_consensus.cc:357] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.158095 14281 raft_consensus.cc:383] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.158254 14281 raft_consensus.cc:738] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.158623 14281 consensus_queue.cc:260] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.160631 14277 ts_tablet_manager.cc:1428] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 08:12:38.162362 14281 ts_tablet_manager.cc:1428] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 08:12:38.166123 14278 tablet_bootstrap.cc:492] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:38.166296 14278 ts_tablet_manager.cc:1397] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250903 08:12:38.168397 14278 raft_consensus.cc:357] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.168579 14278 raft_consensus.cc:383] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.168684 14278 raft_consensus.cc:738] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.168926 14278 consensus_queue.cc:260] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.170651 14278 ts_tablet_manager.cc:1428] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 08:12:38.185565 14309 raft_consensus.cc:491] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.185825 14309 raft_consensus.cc:513] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.186573 14309 leader_election.cc:290] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:38.187148 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da45d19c545a45c2abff9c05d8c0b293" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.187287 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da45d19c545a45c2abff9c05d8c0b293" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.187413 14028 raft_consensus.cc:2466] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.187534 13890 raft_consensus.cc:2466] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.187951 14097 leader_election.cc:304] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.188306 14309 raft_consensus.cc:2802] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.188436 14309 raft_consensus.cc:491] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.188550 14309 raft_consensus.cc:3058] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.191334 14309 raft_consensus.cc:513] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.192682 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da45d19c545a45c2abff9c05d8c0b293" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.192943 14028 raft_consensus.cc:3058] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.195899 14028 raft_consensus.cc:2466] T da45d19c545a45c2abff9c05d8c0b293 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.197049 14097 leader_election.cc:304] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.197445 14283 raft_consensus.cc:2802] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.197633 14283 raft_consensus.cc:695] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [term 1 LEADER]: Becoming Leader. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Running, Role: LEADER
I20250903 08:12:38.198122 14283 consensus_queue.cc:237] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.198428 14309 leader_election.cc:290] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:38.199736 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da45d19c545a45c2abff9c05d8c0b293" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.199961 13890 raft_consensus.cc:3058] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.203569 13890 raft_consensus.cc:2466] T da45d19c545a45c2abff9c05d8c0b293 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.204087 14283 raft_consensus.cc:491] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.204264 14283 raft_consensus.cc:513] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.205515 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43782057c894c77ad657f67deb11a71" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.205634 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43782057c894c77ad657f67deb11a71" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.205758 13889 raft_consensus.cc:2466] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.205911 14028 raft_consensus.cc:2466] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:12:38.206387 14098 leader_election.cc:304] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.206612 14283 leader_election.cc:290] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.206812 14309 raft_consensus.cc:2802] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.206907 14309 raft_consensus.cc:491] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.207007 14309 raft_consensus.cc:3058] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.210018 14309 raft_consensus.cc:513] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.211378 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43782057c894c77ad657f67deb11a71" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.212033 14028 raft_consensus.cc:3058] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.213941 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43782057c894c77ad657f67deb11a71" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.214182 13890 raft_consensus.cc:3058] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.216404 14028 raft_consensus.cc:2466] T a43782057c894c77ad657f67deb11a71 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.216634 14309 leader_election.cc:290] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.217001 13890 raft_consensus.cc:2466] T a43782057c894c77ad657f67deb11a71 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:12:38.217237 14097 leader_election.cc:304] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.217936 14309 raft_consensus.cc:2802] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.218140 14309 raft_consensus.cc:695] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [term 1 LEADER]: Becoming Leader. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Running, Role: LEADER
I20250903 08:12:38.218508 14309 consensus_queue.cc:237] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.261394 14284 raft_consensus.cc:491] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.261623 14284 raft_consensus.cc:513] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.263628 14284 leader_election.cc:290] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.270020 14286 raft_consensus.cc:491] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.270273 14286 raft_consensus.cc:513] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.272425 14286 leader_election.cc:290] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.293416 14284 raft_consensus.cc:491] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.293663 14284 raft_consensus.cc:513] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.294410 14284 leader_election.cc:290] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.296250 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65826010a9b482faf67a7120247eab1" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.296491 14028 raft_consensus.cc:2466] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.297040 13826 leader_election.cc:304] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.297577 14286 raft_consensus.cc:2802] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.297745 14286 raft_consensus.cc:491] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.297858 14286 raft_consensus.cc:3058] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.300796 14286 raft_consensus.cc:513] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.302076 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65826010a9b482faf67a7120247eab1" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.302330 14028 raft_consensus.cc:3058] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.305326 14028 raft_consensus.cc:2466] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.306521 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65826010a9b482faf67a7120247eab1" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.306962 14162 raft_consensus.cc:2466] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.308115 13826 leader_election.cc:304] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.309202 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65826010a9b482faf67a7120247eab1" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:12:38.309451 14162 raft_consensus.cc:3058] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.312412 14162 raft_consensus.cc:2466] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.313122 14286 leader_election.cc:290] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.313417 14286 raft_consensus.cc:2802] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.313597 14286 raft_consensus.cc:695] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Becoming Leader. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Running, Role: LEADER
I20250903 08:12:38.314057 14286 consensus_queue.cc:237] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.325416 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73649395ff784d5cb8ffff12c3e45dbe" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.325742 13890 raft_consensus.cc:2466] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:12:38.326434 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.326805 13890 raft_consensus.cc:2391] T 4a67d22967884a02a6a89937065d2569 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 668be13bcd854aa8afa97d2bbaf067a7 in current term 1: Already voted for candidate 0b68b07307444fdebbee0bff3fb79869 in this term.
I20250903 08:12:38.327596 13964 leader_election.cc:304] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.328083 14284 raft_consensus.cc:2802] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.328219 14284 raft_consensus.cc:491] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.328338 14284 raft_consensus.cc:3058] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.331347 14284 raft_consensus.cc:513] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.332423 14284 leader_election.cc:290] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.333564 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73649395ff784d5cb8ffff12c3e45dbe" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.333767 13890 raft_consensus.cc:3058] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.336707 13890 raft_consensus.cc:2466] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:12:38.337304 13964 leader_election.cc:304] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.337651 14284 raft_consensus.cc:2802] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.349926 14161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a67d22967884a02a6a89937065d2569" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.350363 14284 raft_consensus.cc:695] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Becoming Leader. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Running, Role: LEADER
I20250903 08:12:38.350919 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73649395ff784d5cb8ffff12c3e45dbe" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:12:38.351243 14160 raft_consensus.cc:3058] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.351799 14284 consensus_queue.cc:237] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.354765 13962 leader_election.cc:304] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7; no voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d
I20250903 08:12:38.355124 14327 raft_consensus.cc:3058] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.355664 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73649395ff784d5cb8ffff12c3e45dbe" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.357180 14160 raft_consensus.cc:2466] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:12:38.358284 14327 raft_consensus.cc:2747] T 4a67d22967884a02a6a89937065d2569 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
08:12:38.367 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_3f66d15db9bc4be7bf75984378182586 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250903 08:12:38.369637 13624 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=ae5dfd31fd7c446cabac27c4668ab3d9]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_3f66d15db9bc4be7bf75984378182586 already exists)
I20250903 08:12:38.370137 13624 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34214 (request call id 19) took 1058 ms. Trace:
I20250903 08:12:38.370297 13624 rpcz_store.cc:276] 0903 08:12:37.311183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 08:12:37.311803 (+ 620us) service_pool.cc:225] Handling call
0903 08:12:37.515776 (+203973us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 08:12:37.516151 (+ 375us) catalog_manager.cc:2122] Created new table and tablet info
0903 08:12:38.370091 (+853940us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":563364,"Hive Metastore.run_cpu_time_us":865,"Hive Metastore.run_wall_time_us":289778,"spinlock_wait_cycles":19200}
I20250903 08:12:38.373533 13624 catalog_manager.cc:5582] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67). New cstate: current_term: 1 leader_uuid: "0b68b07307444fdebbee0bff3fb79869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: HEALTHY } } }
I20250903 08:12:38.385954 13624 catalog_manager.cc:5582] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d reported cstate change: term changed from 0 to 1, leader changed from <none> to 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65). New cstate: current_term: 1 leader_uuid: "70c467196f6846b78b5a7c7ac8a2787d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: UNKNOWN } } }
I20250903 08:12:38.396908 13624 catalog_manager.cc:5582] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67). New cstate: current_term: 1 leader_uuid: "0b68b07307444fdebbee0bff3fb79869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: HEALTHY } } }
I20250903 08:12:38.397552 13624 catalog_manager.cc:5582] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67). New cstate: current_term: 1 leader_uuid: "0b68b07307444fdebbee0bff3fb79869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: HEALTHY } } }
I20250903 08:12:38.407640 13624 catalog_manager.cc:5582] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66). New cstate: current_term: 1 leader_uuid: "668be13bcd854aa8afa97d2bbaf067a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } health_report { overall_health: UNKNOWN } } }
I20250903 08:12:38.415185 13630 catalog_manager.cc:2149] created HMS catalog entry for table [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:12:38.415524 13624 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_b21f9eea82374d75a52c4b6787afdf48" } new_table_name: "default.Table_bde72d06420747c19ba769641487520e" modify_external_catalogs: true
I20250903 08:12:38.420152 13630 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34214 (request call id 13) took 1109 ms. Trace:
I20250903 08:12:38.420331 13630 rpcz_store.cc:276] 0903 08:12:37.310477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 08:12:37.310631 (+ 154us) service_pool.cc:225] Handling call
0903 08:12:37.516634 (+206003us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 08:12:37.517066 (+ 432us) catalog_manager.cc:2122] Created new table and tablet info
0903 08:12:38.415124 (+898058us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 08:12:38.419819 (+ 4695us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 08:12:38.419953 (+ 134us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 08:12:38.420086 (+ 133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 08:12:38.416053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 08:12:38.416179 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0903 08:12:38.416183 (+ 4us) write_op.cc:435] Acquired schema lock
0903 08:12:38.416185 (+ 2us) tablet.cc:660] Decoding operations
0903 08:12:38.416258 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0903 08:12:38.416269 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0903 08:12:38.416272 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0903 08:12:38.416331 (+ 59us) tablet.cc:699] Row locks acquired
0903 08:12:38.416333 (+ 2us) write_op.cc:260] PREPARE: finished
0903 08:12:38.416362 (+ 29us) write_op.cc:270] Start()
0903 08:12:38.416386 (+ 24us) write_op.cc:276] Timestamp: P: 1756887158416345 usec, L: 0
0903 08:12:38.416389 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 08:12:38.416549 (+ 160us) log.cc:844] Serialized 432 byte log entry
0903 08:12:38.419132 (+ 2583us) op_driver.cc:464] REPLICATION: finished
0903 08:12:38.419297 (+ 165us) write_op.cc:301] APPLY: starting
0903 08:12:38.419311 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0903 08:12:38.419366 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0903 08:12:38.419419 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 08:12:38.419500 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 08:12:38.419529 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 08:12:38.419534 (+ 5us) write_op.cc:312] APPLY: finished
0903 08:12:38.419598 (+ 64us) log.cc:844] Serialized 28 byte log entry
0903 08:12:38.419675 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0903 08:12:38.419708 (+ 33us) write_op.cc:454] Released schema lock
0903 08:12:38.419715 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 08:12:38.419797 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0903 08:12:38.419801 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":852311,"Hive Metastore.run_cpu_time_us":311,"Hive Metastore.run_wall_time_us":45548,"spinlock_wait_cycles":135296,"child_traces":[["op",{"apply.queue_time_us":113,"apply.run_cpu_time_us":489,"apply.run_wall_time_us":488,"num_ops":2,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":619,"prepare.run_wall_time_us":619,"raft.queue_time_us":426,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":242,"replication_time_us":2713}]]}
I20250903 08:12:38.421594 13630 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34214:
name: "default.Table_6221cb658b3f49fb8d60a02df2b7a834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:38.427868 14008 tablet_service.cc:1468] Processing CreateTablet for tablet 879297b2be2c4717a747a95140139e6f (DEFAULT_TABLE table=default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.428169 14141 tablet_service.cc:1468] Processing CreateTablet for tablet 879297b2be2c4717a747a95140139e6f (DEFAULT_TABLE table=default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.428400 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879297b2be2c4717a747a95140139e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.428575 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879297b2be2c4717a747a95140139e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.432312 13871 tablet_service.cc:1468] Processing CreateTablet for tablet 879297b2be2c4717a747a95140139e6f (DEFAULT_TABLE table=default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.432754 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879297b2be2c4717a747a95140139e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.437908 14278 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:38.439287 14278 tablet_bootstrap.cc:654] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.443378 14277 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:38.444375 14281 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.445065 14277 tablet_bootstrap.cc:654] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.446211 14281 tablet_bootstrap.cc:654] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.451915 14278 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:38.452172 14278 ts_tablet_manager.cc:1397] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250903 08:12:38.453239 14278 raft_consensus.cc:357] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.453540 14278 raft_consensus.cc:383] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.453859 14278 raft_consensus.cc:738] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.454216 14278 consensus_queue.cc:260] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.455662 14278 ts_tablet_manager.cc:1428] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:12:38.462589 14281 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.462841 14281 ts_tablet_manager.cc:1397] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.009s
I20250903 08:12:38.464099 14281 raft_consensus.cc:357] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.464360 14281 raft_consensus.cc:383] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.464450 14281 raft_consensus.cc:738] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.464805 14281 consensus_queue.cc:260] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.467969 14281 ts_tablet_manager.cc:1428] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250903 08:12:38.470927 14277 tablet_bootstrap.cc:492] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.471133 14277 ts_tablet_manager.cc:1397] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.006s
I20250903 08:12:38.471786 14277 raft_consensus.cc:357] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.471954 14277 raft_consensus.cc:383] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.472026 14277 raft_consensus.cc:738] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.472268 14277 consensus_queue.cc:260] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.472920 14277 ts_tablet_manager.cc:1428] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 08:12:38.500804 13628 catalog_manager.cc:2149] created HMS catalog entry for table [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:12:38.508595 13628 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34214 (request call id 15) took 1197 ms. Trace:
I20250903 08:12:38.508775 13628 rpcz_store.cc:276] 0903 08:12:37.310759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 08:12:37.311758 (+ 999us) service_pool.cc:225] Handling call
0903 08:12:37.517583 (+205825us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 08:12:37.517991 (+ 408us) catalog_manager.cc:2122] Created new table and tablet info
0903 08:12:38.500739 (+982748us) catalog_manager.cc:2148] Created new table in HMS catalog
0903 08:12:38.508274 (+ 7535us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 08:12:38.508419 (+ 145us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 08:12:38.508544 (+ 125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 08:12:38.501924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 08:12:38.502066 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0903 08:12:38.502073 (+ 7us) write_op.cc:435] Acquired schema lock
0903 08:12:38.502076 (+ 3us) tablet.cc:660] Decoding operations
0903 08:12:38.502200 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0903 08:12:38.502216 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 08:12:38.502310 (+ 94us) tablet.cc:683] Acquiring locks for 2 operations
0903 08:12:38.502407 (+ 97us) tablet.cc:699] Row locks acquired
0903 08:12:38.502410 (+ 3us) write_op.cc:260] PREPARE: finished
0903 08:12:38.502462 (+ 52us) write_op.cc:270] Start()
0903 08:12:38.502507 (+ 45us) write_op.cc:276] Timestamp: P: 1756887158502432 usec, L: 0
0903 08:12:38.502511 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 08:12:38.502753 (+ 242us) log.cc:844] Serialized 432 byte log entry
0903 08:12:38.507007 (+ 4254us) op_driver.cc:464] REPLICATION: finished
0903 08:12:38.507148 (+ 141us) write_op.cc:301] APPLY: starting
0903 08:12:38.507167 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 08:12:38.507218 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0903 08:12:38.507256 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 08:12:38.507366 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 08:12:38.507406 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 08:12:38.507415 (+ 9us) write_op.cc:312] APPLY: finished
0903 08:12:38.507508 (+ 93us) log.cc:844] Serialized 28 byte log entry
0903 08:12:38.507592 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0903 08:12:38.507618 (+ 26us) write_op.cc:454] Released schema lock
0903 08:12:38.507625 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 08:12:38.507717 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0903 08:12:38.507721 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":896975,"Hive Metastore.run_cpu_time_us":321,"Hive Metastore.run_wall_time_us":85432,"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":557,"apply.run_wall_time_us":557,"num_ops":2,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":884,"prepare.run_wall_time_us":971,"raft.queue_time_us":1391,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":353,"replication_time_us":4463}]]}
F20250903 08:12:38.511514 13793 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f684ce4ddcd google::LogMessage::Fail() at ??:0
@ 0x7f684ce51b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f684ce4d7cc google::LogMessage::Flush() at ??:0
@ 0x7f684ce4da11 google::LogMessage::~LogMessage() at ??:0
I20250903 08:12:38.523818 14008 tablet_service.cc:1468] Processing CreateTablet for tablet d1bd9a796f9947acb875272cf4e22c21 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.523818 13871 tablet_service.cc:1468] Processing CreateTablet for tablet d1bd9a796f9947acb875272cf4e22c21 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.524277 13871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bd9a796f9947acb875272cf4e22c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.524281 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bd9a796f9947acb875272cf4e22c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.531267 14141 tablet_service.cc:1468] Processing CreateTablet for tablet d1bd9a796f9947acb875272cf4e22c21 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:12:38.533381 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bd9a796f9947acb875272cf4e22c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:12:38.536720 14277 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:12:38.537711 14286 raft_consensus.cc:491] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.537905 14286 raft_consensus.cc:513] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.539247 14277 tablet_bootstrap.cc:654] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.539842 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdfe060848fc4e9d9534e539f39137da" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.540063 14028 raft_consensus.cc:2466] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.540657 13826 leader_election.cc:304] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.541069 14325 raft_consensus.cc:2802] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.541172 14325 raft_consensus.cc:491] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.541321 14325 raft_consensus.cc:3058] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.545336 14325 raft_consensus.cc:513] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.545861 14286 leader_election.cc:290] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.546891 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdfe060848fc4e9d9534e539f39137da" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.547178 14028 raft_consensus.cc:3058] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.549485 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdfe060848fc4e9d9534e539f39137da" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.549814 14160 raft_consensus.cc:2466] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.551213 14028 raft_consensus.cc:2466] T bdfe060848fc4e9d9534e539f39137da P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.551796 13826 leader_election.cc:304] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.552201 14286 raft_consensus.cc:2802] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.552388 14325 leader_election.cc:290] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.553192 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bdfe060848fc4e9d9534e539f39137da" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:12:38.553498 14160 raft_consensus.cc:3058] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.556771 14160 raft_consensus.cc:2466] T bdfe060848fc4e9d9534e539f39137da P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.557744 14286 raft_consensus.cc:695] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Becoming Leader. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Running, Role: LEADER
I20250903 08:12:38.557742 13628 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb" } modify_external_catalogs: true
I20250903 08:12:38.558075 14286 consensus_queue.cc:237] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.562072 14281 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:12:38.564070 14281 tablet_bootstrap.cc:654] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:12:38.566829 13628 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34214:
table { table_name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb" } modify_external_catalogs: true
I20250903 08:12:38.568094 14278 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:12:38.570384 14278 tablet_bootstrap.cc:654] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
W20250903 08:12:38.570890 14298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:38.571281 14298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:38.571413 14298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 08:12:38.572104 14281 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:12:38.572676 14281 ts_tablet_manager.cc:1397] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250903 08:12:38.573881 14281 raft_consensus.cc:357] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.574142 14281 raft_consensus.cc:383] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.574256 14281 raft_consensus.cc:738] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.574544 14281 consensus_queue.cc:260] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.575261 14281 ts_tablet_manager.cc:1428] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
@ 0x7f68632907b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250903 08:12:38.582806 14278 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:12:38.582978 14278 ts_tablet_manager.cc:1397] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20250903 08:12:38.583794 14278 raft_consensus.cc:357] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.583978 14278 raft_consensus.cc:383] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.584049 14278 raft_consensus.cc:738] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
@ 0x7f686328f061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 08:12:38.584292 14278 consensus_queue.cc:260] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.584798 14278 ts_tablet_manager.cc:1428] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250903 08:12:38.585136 14298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:38.585237 14298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:38.585382 14298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:38.585449 14298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:38.588414 14277 tablet_bootstrap.cc:492] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:12:38.588588 14277 ts_tablet_manager.cc:1397] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.033s
I20250903 08:12:38.589644 14277 raft_consensus.cc:357] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.589891 14277 raft_consensus.cc:383] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:12:38.589984 14277 raft_consensus.cc:738] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.590236 14277 consensus_queue.cc:260] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.591804 14277 ts_tablet_manager.cc:1428] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
@ 0x7f6863296cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f6863296b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 08:12:38.602490 14298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
@ 0x7f684fbb62b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6852fda6db start_thread at ??:0
I20250903 08:12:38.604816 14298 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
@ 0x7f684bcfe71f clone at ??:0
I20250903 08:12:38.607861 14298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:38.624994 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:38.626154 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:38.641466 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:38.642046 14298 server_base.cc:1047] running on GCE node
I20250903 08:12:38.643038 14298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:38.644868 14298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:38.646131 14298 hybrid_clock.cc:648] HybridClock initialized: now 1756887158646069 us; error 54 us; skew 500 ppm
I20250903 08:12:38.646584 14298 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:38.649597 14286 raft_consensus.cc:491] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.649722 14298 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:38.649808 14286 raft_consensus.cc:513] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.650559 14286 leader_election.cc:290] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.650807 14298 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:38.650961 14298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:38.651196 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bd9a796f9947acb875272cf4e22c21" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:12:38.651338 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bd9a796f9947acb875272cf4e22c21" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.651446 14028 raft_consensus.cc:2466] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.651538 14160 raft_consensus.cc:2466] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:12:38.651989 13825 leader_election.cc:304] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.652308 14286 raft_consensus.cc:2802] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.652410 14286 raft_consensus.cc:491] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.652509 14286 raft_consensus.cc:3058] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.655133 14286 raft_consensus.cc:513] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.656376 14286 leader_election.cc:290] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.656342 14028 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bd9a796f9947acb875272cf4e22c21" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:12:38.656486 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1bd9a796f9947acb875272cf4e22c21" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:12:38.656567 14028 raft_consensus.cc:3058] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.656692 14160 raft_consensus.cc:3058] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.659443 14160 raft_consensus.cc:2466] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.659438 14028 raft_consensus.cc:2466] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:12:38.659983 13825 leader_election.cc:304] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.660326 14286 raft_consensus.cc:2802] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.660571 14286 raft_consensus.cc:695] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Becoming Leader. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Running, Role: LEADER
I20250903 08:12:38.660916 14286 consensus_queue.cc:237] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.662173 14298 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.002s sys 0.005s
I20250903 08:12:38.670017 14363 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:38.672086 14298 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20250903 08:12:38.672302 14298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:38.674283 14298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:38.687855 14298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:38.688853 14298 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:38.689141 14298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:38.710920 14298 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:38.711766 14414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:38.712841 14298 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:38.719756 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14298
I20250903 08:12:38.722872 14415 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:38.728124 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:38.732012 14284 raft_consensus.cc:491] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:38.732213 14284 raft_consensus.cc:513] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.733048 14284 leader_election.cc:290] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.733937 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "879297b2be2c4717a747a95140139e6f" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:12:38.734030 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "879297b2be2c4717a747a95140139e6f" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:12:38.734192 13890 raft_consensus.cc:2466] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:12:38.734246 14160 raft_consensus.cc:2466] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:12:38.734678 13962 leader_election.cc:304] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:12:38.735195 14284 raft_consensus.cc:2802] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:12:38.735303 14284 raft_consensus.cc:491] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:38.735414 14284 raft_consensus.cc:3058] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.738128 14284 raft_consensus.cc:513] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.739224 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "879297b2be2c4717a747a95140139e6f" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:12:38.739470 13890 raft_consensus.cc:3058] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.740012 14415 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:38.741703 14284 leader_election.cc:290] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:38.742750 14160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "879297b2be2c4717a747a95140139e6f" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:12:38.742930 13890 raft_consensus.cc:2466] T 879297b2be2c4717a747a95140139e6f P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:12:38.743000 14160 raft_consensus.cc:3058] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:12:38.743417 13964 leader_election.cc:304] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:12:38.743728 14284 raft_consensus.cc:2802] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:12:38.743856 14284 raft_consensus.cc:695] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Becoming Leader. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Running, Role: LEADER
I20250903 08:12:38.744138 14284 consensus_queue.cc:237] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:12:38.745780 14160 raft_consensus.cc:2466] T 879297b2be2c4717a747a95140139e6f P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:12:38.753803 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:38.755707 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:12:38.756356 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:12:38.764840 14415 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:38.765918 14415 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:38.766778 14415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:38.767810 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:38.768239 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:38.769200 14415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:38.784130 14434 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:38.784257 14434 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:12:38.788793 14434 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:12:38.792047 14434 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250903 08:12:38.864631 14215 connection.cc:537] client connection to 127.12.250.125:34047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:38.865516 14215 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.125:34047)
W20250903 08:12:38.865998 14215 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.250.125:34047)
W20250903 08:12:38.866641 14215 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.250.125:34047)
W20250903 08:12:38.867800 14215 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.250.125:34047)
W20250903 08:12:38.869302 14436 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:12:38.797515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.250.125:34047 (local address 127.0.0.1:34228)
0903 08:12:38.797803 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0903 08:12:38.797824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0903 08:12:38.797963 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:12:38.867931 (+ 69968us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173}
W20250903 08:12:38.870946 14264 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:12:38.867099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.250.125:34047 (local address 127.12.250.66:41697)
0903 08:12:38.867352 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0903 08:12:38.867373 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0903 08:12:38.867504 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0903 08:12:38.870077 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127}
I20250903 08:12:38.870645 14298 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250903 08:12:38.879426 14298 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:38.942380 14309 consensus_queue.cc:1035] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:38.949667 14380 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:41767; Asking this server to re-register.
I20250903 08:12:38.951777 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:38.954813 14379 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:38.959455 14309 consensus_queue.cc:1035] T 4a67d22967884a02a6a89937065d2569 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:38.975029 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:38.975476 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:38.975726 14208 heartbeater.cc:499] Master 127.12.250.126:42877 was elected leader, sending a full tablet report...
I20250903 08:12:38.977077 14379 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:42195; Asking this server to re-register.
I20250903 08:12:38.977921 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:38.977993 14379 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:33005; Asking this server to re-register.
I20250903 08:12:38.979138 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:38.979956 14379 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:38.981526 14379 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:39.032667 14449 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dba49201780c41ebb58a4c8c0ae7bb1e)
I20250903 08:12:39.032864 14449 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:39.033682 14449 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
W20250903 08:12:39.036481 13686 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:39.042742 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
I20250903 08:12:39.043349 14390 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9eecf53ccf4633bc701ff21e64ab0b in term 1.
I20250903 08:12:39.043901 13685 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:12:39.044207 14449 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 08:12:39.044340 14449 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Starting leader election (detected failure of leader dba49201780c41ebb58a4c8c0ae7bb1e)
I20250903 08:12:39.044456 14449 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:12:39.046496 14449 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:39.047262 14449 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 election: Requested vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
I20250903 08:12:39.047727 14390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611"
I20250903 08:12:39.047953 14390 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 1 FOLLOWER]: Advancing to term 3
W20250903 08:12:39.048776 13686 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:39.052318 14390 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9eecf53ccf4633bc701ff21e64ab0b in term 3.
I20250903 08:12:39.052862 13685 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:12:39.053195 14449 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Leader election won for term 3
I20250903 08:12:39.053757 14449 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 LEADER]: Becoming Leader. State: Replica: ff9eecf53ccf4633bc701ff21e64ab0b, State: Running, Role: LEADER
I20250903 08:12:39.054183 14449 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:39.055670 14453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9eecf53ccf4633bc701ff21e64ab0b. Latest consensus state: current_term: 3 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:39.056041 14453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:39.056867 14454 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:12:39.060544 14454 catalog_manager.cc:671] Loaded metadata for table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:12:39.061197 14454 catalog_manager.cc:671] Loaded metadata for table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:12:39.061731 14454 catalog_manager.cc:671] Loaded metadata for table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]
I20250903 08:12:39.062247 14454 catalog_manager.cc:671] Loaded metadata for table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:12:39.062724 14454 catalog_manager.cc:671] Loaded metadata for table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:12:39.063192 14454 catalog_manager.cc:671] Loaded metadata for table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:12:39.063669 14454 catalog_manager.cc:671] Loaded metadata for table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:12:39.064122 14454 catalog_manager.cc:671] Loaded metadata for table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:12:39.064605 14454 catalog_manager.cc:671] Loaded metadata for table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:12:39.067296 14454 tablet_loader.cc:96] loaded metadata for tablet 09070dc67d994d26a5c768db60faf96c (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff])
I20250903 08:12:39.067785 14454 tablet_loader.cc:96] loaded metadata for tablet 4a67d22967884a02a6a89937065d2569 (table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995])
I20250903 08:12:39.068490 14454 tablet_loader.cc:96] loaded metadata for tablet 73649395ff784d5cb8ffff12c3e45dbe (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431])
I20250903 08:12:39.068907 14454 tablet_loader.cc:96] loaded metadata for tablet 879297b2be2c4717a747a95140139e6f (table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05])
I20250903 08:12:39.069372 14454 tablet_loader.cc:96] loaded metadata for tablet a43782057c894c77ad657f67deb11a71 (table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a])
I20250903 08:12:39.069792 14454 tablet_loader.cc:96] loaded metadata for tablet bdfe060848fc4e9d9534e539f39137da (table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b])
I20250903 08:12:39.070188 14454 tablet_loader.cc:96] loaded metadata for tablet d1bd9a796f9947acb875272cf4e22c21 (table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff])
I20250903 08:12:39.070703 14454 tablet_loader.cc:96] loaded metadata for tablet da45d19c545a45c2abff9c05d8c0b293 (table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12])
I20250903 08:12:39.071382 14454 tablet_loader.cc:96] loaded metadata for tablet f65826010a9b482faf67a7120247eab1 (table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5])
I20250903 08:12:39.071724 14454 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:12:39.073240 14454 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:39.073366 14454 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:12:39.075269 14454 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:12:39.076675 14454 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Loaded TSK: 0
I20250903 08:12:39.077930 14454 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:12:39.079254 14454 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:12:39.080598 14454 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:12:39.097913 13701 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39848:
name: "default.Table_c88874dd846641f19e00bc8d13e3377b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:39.097988 13700 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39848:
name: "default.Table_58f55395478b4968a5f5e04b0139ac01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:39.099920 13694 master_service.cc:948] ReplaceTablet: received request to replace tablet 09070dc67d994d26a5c768db60faf96c from {username='slave'} at 127.0.0.1:39848
I20250903 08:12:39.099800 13695 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39848:
name: "default.Table_3b5331a45947402b9deafb9d63b55b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:39.098768 13699 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_fc56d0f82c054a9facf698e29514b8d4" } modify_external_catalogs: true
I20250903 08:12:39.100387 13699 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_fc56d0f82c054a9facf698e29514b8d4" } modify_external_catalogs: true
I20250903 08:12:39.098830 13698 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58" } modify_external_catalogs: true
I20250903 08:12:39.100739 13698 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58" } modify_external_catalogs: true
I20250903 08:12:39.099020 13696 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39848:
name: "default.Table_1fb5beb8dbab4781ab4c84eb8414a936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:39.101512 13692 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_feb51f00a18446ed842cbf54db2f3e06" } modify_external_catalogs: true
I20250903 08:12:39.101737 13692 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39848:
table { table_name: "default.Table_feb51f00a18446ed842cbf54db2f3e06" } modify_external_catalogs: true
I20250903 08:12:39.101233 13693 master_service.cc:948] ReplaceTablet: received request to replace tablet a43782057c894c77ad657f67deb11a71 from {username='slave'} at 127.0.0.1:39848
I20250903 08:12:39.099182 13697 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39848:
name: "default.Table_6a11e40b85c9416abb127a1c1ecf1e0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:39.107710 14390 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Refusing update from remote peer ff9eecf53ccf4633bc701ff21e64ab0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250903 08:12:39.108778 14453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 4, Time since last communication: 0.000s
W20250903 08:12:39.112318 13686 consensus_peers.cc:489] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b -> Peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Couldn't send request to peer dba49201780c41ebb58a4c8c0ae7bb1e. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20250903 08:12:39.113770 13775 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff1dca7adcd google::LogMessage::Fail() at ??:0
@ 0x7ff1dca7eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff1dca7a7cc google::LogMessage::Flush() at ??:0
@ 0x7ff1dca7aa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff1f2ebd7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ff1f2ebc061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff1f2ec3cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ff1f2ec3b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7ff1df7e32b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1e2c076db start_thread at ??:0
I20250903 08:12:39.158452 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9eecf53ccf4633bc701ff21e64ab0b. Latest consensus state: current_term: 3 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:39.158780 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:39.158468 14459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:39.158941 14459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7ff1db92b71f clone at ??:0
I20250903 08:12:39.175971 14309 consensus_queue.cc:1035] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.176998 14456 mvcc.cc:204] Tried to move back new op lower bound from 7196209801273024512 to 7196209796284772352. Current Snapshot: MvccSnapshot[applied={T|T < 7196209801273024512 or (T in {7196209801273024512})}]
I20250903 08:12:39.192052 14441 consensus_queue.cc:1035] T a43782057c894c77ad657f67deb11a71 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.220996 14284 consensus_queue.cc:1035] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.227290 14472 consensus_queue.cc:1035] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.233412 14327 consensus_queue.cc:1035] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.240942 14286 consensus_queue.cc:1035] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.245729 14472 consensus_queue.cc:1035] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.263053 14487 consensus_queue.cc:1035] T 09070dc67d994d26a5c768db60faf96c P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.270081 14472 consensus_queue.cc:1035] T da45d19c545a45c2abff9c05d8c0b293 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.301822 14286 consensus_queue.cc:1035] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 08:12:39.413128 13961 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:39.413157 14095 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:39.414732 13823 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 08:12:39.503633 14327 consensus_queue.cc:1035] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.511075 14494 consensus_queue.cc:1035] T 879297b2be2c4717a747a95140139e6f P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.613446 14459 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:12:39.613700 14459 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:39.615558 14459 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:39.616532 14365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:39.617816 14364 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:39.618018 14365 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:39.618197 14365 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:39.618554 14459 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:39.678993 14286 consensus_queue.cc:1035] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.684826 14480 consensus_queue.cc:1035] T bdfe060848fc4e9d9534e539f39137da P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.754719 14286 consensus_queue.cc:1035] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:39.766026 14286 consensus_queue.cc:1035] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:12:40.329856 14513 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:12:40.330076 14513 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:40.330826 14513 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:40.332624 14365 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:40.332795 14364 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:40.332939 14364 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:40.333173 14513 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:40.998417 13289 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu pid 14298
I20250903 08:12:41.009006 14410 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 08:12:41.120755 13289 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu with pid 14298
I20250903 08:12:42.096664 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:12:42.452068 14542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:42.452467 14542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:42.452603 14542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:42.461315 14542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:42.461452 14542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:42.461568 14542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:42.461637 14542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:42.471782 14542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:42.473567 14542 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:42.475555 14542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:42.484239 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:42.484316 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:42.484531 14555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:42.486349 14542 server_base.cc:1047] running on GCE node
I20250903 08:12:42.487195 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:42.488751 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:42.489959 14542 hybrid_clock.cc:648] HybridClock initialized: now 1756887162489911 us; error 42 us; skew 500 ppm
I20250903 08:12:42.490334 14542 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:42.492683 14542 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:42.493522 14542 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:42.493635 14542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:42.499394 14542 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20250903 08:12:42.502559 14561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:42.504209 14542 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:12:42.504411 14542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:42.505507 14542 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:42.526412 14542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:42.527410 14542 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:42.527711 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:42.548457 14542 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:42.548465 14613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:42.550563 14542 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:42.557113 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14542
I20250903 08:12:42.557747 14614 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:42.562578 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:42.573375 14614 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:42.601902 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250903 08:12:42.602631 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:12:42.608023 14614 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:42.611714 14614 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:42.612444 14614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:42.613204 14619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:42.613514 14619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:42.614353 14614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:42.632141 14633 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:42.632405 14633 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:12:42.638859 14633 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:12:42.644450 14633 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:42.670132 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:42.667505 14542 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:12:42.672684 14542 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:42.673866 14579 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:42539; Asking this server to re-register.
I20250903 08:12:42.674636 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:42.676425 14579 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:42.771003 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:42.773380 14579 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:46723; Asking this server to re-register.
I20250903 08:12:42.774183 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:42.776018 14579 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:42.935770 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:42.938015 14579 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:57969; Asking this server to re-register.
I20250903 08:12:42.938781 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:42.940415 14579 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:43.176414 14643 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:43.176733 14643 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:43.179106 14643 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:43.180104 14563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:43.181974 14563 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:43.182178 14562 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:43.182325 14562 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:43.182714 14643 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:43.843932 14650 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:43.844116 14650 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:43.844857 14650 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:43.847086 14563 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:43.847297 14562 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:43.847419 14562 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:43.847664 14650 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:44.863605 14661 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:44.863816 14661 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:44.864454 14661 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:44.867051 14562 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:44.867228 14563 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:44.867472 14563 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:44.867837 14661 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:44.961076 13289 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu pid 14542
I20250903 08:12:44.968096 14609 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 08:12:45.074553 13289 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu with pid 14542
I20250903 08:12:45.947299 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:12:46.300450 14677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:46.300796 14677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:46.300923 14677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:46.309475 14677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:46.309585 14677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:46.309659 14677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:46.309753 14677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:46.320019 14677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:46.321830 14677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:46.323734 14677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:46.332307 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:46.332306 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:46.332943 14686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:46.333747 14677 server_base.cc:1047] running on GCE node
I20250903 08:12:46.334515 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:46.336158 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:46.337360 14677 hybrid_clock.cc:648] HybridClock initialized: now 1756887166337300 us; error 57 us; skew 500 ppm
I20250903 08:12:46.337695 14677 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:46.340242 14677 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:46.341034 14677 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:46.341127 14677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:46.347239 14677 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 08:12:46.350401 14692 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:46.352008 14677 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:12:46.352176 14677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:46.352944 14677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:46.368938 14677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:46.369932 14677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:46.370236 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:46.390806 14743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:46.390808 14677 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:46.392637 14677 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:46.397622 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14677
I20250903 08:12:46.399884 14744 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:46.403990 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:46.418598 14744 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:46.448691 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250903 08:12:46.449363 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:12:46.455797 14744 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:46.459699 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:46.459705 14744 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:46.460429 14744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:46.461341 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:46.461622 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:46.462692 14744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:46.477506 14765 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:46.477717 14765 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:12:46.482851 14765 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:12:46.482908 14677 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:12:46.488790 14677 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:46.490173 14765 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:46.699599 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:46.701678 14709 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:41247; Asking this server to re-register.
I20250903 08:12:46.702366 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:46.704186 14709 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:46.796201 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:46.797673 14709 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:47001; Asking this server to re-register.
I20250903 08:12:46.798300 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:46.799677 14709 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:46.896683 14751 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:46.896941 14751 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:46.898844 14751 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:46.899552 14694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:46.901078 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:46.901310 14694 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:46.901505 14694 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:46.901886 14751 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:46.964059 14709 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:46755; Asking this server to re-register.
I20250903 08:12:46.964733 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:46.966188 14709 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:47.506453 14776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:47.506659 14776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:47.507630 14776 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:47.509831 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:47.510067 14694 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:47.510195 14694 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:47.510444 14776 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:48.392612 14787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:48.392822 14787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:48.394167 14787 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:48.395833 14694 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:48.396026 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:48.396139 14693 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:48.396396 14787 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:49.108891 14799 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:49.109063 14799 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:49.109761 14799 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:49.111758 14694 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:49.112015 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:49.112133 14693 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:49.112403 14799 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 08:12:49.818256 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.125:34047
--webserver_interface=127.12.250.125
--webserver_port=36425
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:12:50.155113 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:50.155460 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:50.155582 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:50.164220 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:50.164330 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:50.164397 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:50.164471 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:50.174290 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.125:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.250.125
--webserver_port=36425
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:50.176019 14809 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:50.177906 14809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:50.185881 14817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:50.186707 14818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:50.187453 14820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:50.188071 14809 server_base.cc:1047] running on GCE node
I20250903 08:12:50.188822 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:50.190321 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:50.191522 14809 hybrid_clock.cc:648] HybridClock initialized: now 1756887170191469 us; error 42 us; skew 500 ppm
I20250903 08:12:50.191856 14809 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:50.197559 14809 webserver.cc:480] Webserver started at http://127.12.250.125:36425/ using document root <none> and password file <none>
I20250903 08:12:50.198407 14809 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:50.198552 14809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:50.204589 14809 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20250903 08:12:50.207679 14826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:50.209157 14809 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250903 08:12:50.209381 14809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:12:50.210139 14809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:50.233592 14809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:50.234503 14809 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:50.234803 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:50.255559 14809 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.125:34047
I20250903 08:12:50.255577 14877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.125:34047 every 8 connection(s)
I20250903 08:12:50.257354 14809 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-1/data/info.pb
I20250903 08:12:50.258008 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14809
I20250903 08:12:50.265318 14878 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:50.269465 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Bootstrap starting.
I20250903 08:12:50.280381 14878 log.cc:826] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:50.294909 13938 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:50.317031 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:12:50.317615 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Bootstrap complete.
I20250903 08:12:50.323204 14878 raft_consensus.cc:357] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:50.324178 14878 raft_consensus.cc:738] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba49201780c41ebb58a4c8c0ae7bb1e, State: Initialized, Role: FOLLOWER
I20250903 08:12:50.324986 14878 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:50.326009 14884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:50.326397 14884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:50.327296 14878 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:50.344170 14898 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:50.344318 14898 catalog_manager.cc:1554] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: loading cluster ID for follower catalog manager: success
I20250903 08:12:50.350399 14898 catalog_manager.cc:1576] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: acquiring CA information for follower catalog manager: success
I20250903 08:12:50.355814 14898 catalog_manager.cc:1604] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:50.389660 14809 master_runner.cc:186] Error getting master registration for 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111),
I20250903 08:12:50.394443 14809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:50.457988 14209 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:50.460886 14843 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:52941; Asking this server to re-register.
I20250903 08:12:50.461686 14209 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:50.463465 14843 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:50.570662 14908 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:50.570873 14908 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:50.571609 14908 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:50.574318 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:50.578480 14853 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" is_pre_election: true
I20250903 08:12:50.579039 14853 raft_consensus.cc:2466] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 2.
I20250903 08:12:50.579593 14694 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters: ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:50.579901 14908 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 08:12:50.580042 14908 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:12:50.580139 14908 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:12:50.584887 14908 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:50.585753 14908 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 election: Requested vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:50.586310 14853 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e"
I20250903 08:12:50.586531 14853 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 2 FOLLOWER]: Advancing to term 4
W20250903 08:12:50.587298 14693 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:50.590662 14853 raft_consensus.cc:2466] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 4.
I20250903 08:12:50.591151 14694 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters: ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:50.591547 14908 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Leader election won for term 4
I20250903 08:12:50.591975 14908 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 LEADER]: Becoming Leader. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Running, Role: LEADER
I20250903 08:12:50.592437 14908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:50.595263 14912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:50.595592 14912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:50.689875 14075 heartbeater.cc:344] Connected to a master server at 127.12.250.125:34047
I20250903 08:12:50.692059 14843 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:57271; Asking this server to re-register.
I20250903 08:12:50.692775 14075 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:50.694207 14843 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:50.801719 14843 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:42525; Asking this server to re-register.
I20250903 08:12:50.802414 13938 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:50.803709 14843 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:51.143134 14853 raft_consensus.cc:1273] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Refusing update from remote peer 1e5331161d9e4df39004183d79dc3611: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20250903 08:12:51.143952 14916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 32, Time since last communication: 0.000s
I20250903 08:12:51.149894 14917 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:51.150125 14917 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:51.155679 14927 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:51.156104 14927 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:51.155957 14917 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:51.156319 14917 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:51.159893 14916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:51.160183 14916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:51.160883 14920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:51.161090 14920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
W20250903 08:12:51.162890 14693 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 -> Peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Couldn't send request to peer ff9eecf53ccf4633bc701ff21e64ab0b. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:12:51.241812 14914 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:12:51.244469 14914 catalog_manager.cc:671] Loaded metadata for table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:12:51.244944 14914 catalog_manager.cc:671] Loaded metadata for table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:12:51.245378 14914 catalog_manager.cc:671] Loaded metadata for table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]
I20250903 08:12:51.245733 14914 catalog_manager.cc:671] Loaded metadata for table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:12:51.246063 14914 catalog_manager.cc:671] Loaded metadata for table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:12:51.246400 14914 catalog_manager.cc:671] Loaded metadata for table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:12:51.246706 14914 catalog_manager.cc:671] Loaded metadata for table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:12:51.247026 14914 catalog_manager.cc:671] Loaded metadata for table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:12:51.247330 14914 catalog_manager.cc:671] Loaded metadata for table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:12:51.249336 14914 tablet_loader.cc:96] loaded metadata for tablet 0fc8d373f2f64e9c806440480e9e030d (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff])
I20250903 08:12:51.249722 14914 tablet_loader.cc:96] loaded metadata for tablet 4a67d22967884a02a6a89937065d2569 (table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995])
I20250903 08:12:51.250248 14914 tablet_loader.cc:96] loaded metadata for tablet 73649395ff784d5cb8ffff12c3e45dbe (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431])
I20250903 08:12:51.250543 14914 tablet_loader.cc:96] loaded metadata for tablet 879297b2be2c4717a747a95140139e6f (table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05])
I20250903 08:12:51.250977 14914 tablet_loader.cc:96] loaded metadata for tablet bdfe060848fc4e9d9534e539f39137da (table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b])
I20250903 08:12:51.251214 14914 tablet_loader.cc:96] loaded metadata for tablet bf91b0230f2742beac340a4c622f82ba (table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a])
I20250903 08:12:51.251478 14914 tablet_loader.cc:96] loaded metadata for tablet d1bd9a796f9947acb875272cf4e22c21 (table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff])
I20250903 08:12:51.251735 14914 tablet_loader.cc:96] loaded metadata for tablet da45d19c545a45c2abff9c05d8c0b293 (table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12])
I20250903 08:12:51.252200 14914 tablet_loader.cc:96] loaded metadata for tablet f65826010a9b482faf67a7120247eab1 (table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5])
I20250903 08:12:51.252449 14914 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:12:51.253489 14914 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:51.253577 14914 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:12:51.254921 14914 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:12:51.255890 14914 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Loaded TSK: 0
I20250903 08:12:51.256767 14914 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:12:51.257696 14914 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:12:51.258539 14914 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:12:51.295382 14709 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "default.Table_f7217c183b8a415d90880886722ccbb3" } new_table_name: "default.Table_3a84325d6c704721aaf5a796191718f2" modify_external_catalogs: true
I20250903 08:12:51.295609 14706 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "default.Table_517de34d608c42cc9d4aeafea81f283f" } new_table_name: "default.Table_97201f0afe374761ad6c85589bf4ff69" modify_external_catalogs: true
I20250903 08:12:51.295979 14708 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c" } new_table_name: "default.Table_21c2b2a8c5c14a4b86cfe245eb1e81e9" modify_external_catalogs: true
I20250903 08:12:51.296437 14707 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b" } new_table_name: "default.Table_05bff585c5694de286caf65b794fa796" modify_external_catalogs: true
I20250903 08:12:51.296363 14703 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "default.Table_250ab81379d64db694b3b9013da7f8a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:51.296844 14702 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48816:
table { table_name: "default.Table_b21f9eea82374d75a52c4b6787afdf48" } new_table_name: "default.Table_bde72d06420747c19ba769641487520e" modify_external_catalogs: true
I20250903 08:12:51.297102 14700 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "default.Table_6a11e40b85c9416abb127a1c1ecf1e0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:51.297392 14704 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "default.Table_0ecab4f09e824890b839f9ec45bfa3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:51.297479 14701 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "default.Table_1fb5beb8dbab4781ab4c84eb8414a936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:51.298149 14705 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48816:
name: "default.Table_94faf7a1e44247f6aeaed3759e0b4f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250903 08:12:51.304433 14764 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f79f9df3dcd google::LogMessage::Fail() at ??:0
@ 0x7f79f9df7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f79f9df37cc google::LogMessage::Flush() at ??:0
@ 0x7f79f9df3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7a102367b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f7a10235061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f7a1023ccdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7a1023cb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f79fcb5c2b9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f79fff806db start_thread at ??:0
@ 0x7f79f8ca471f clone at ??:0
W20250903 08:12:51.507556 14095 connection.cc:537] client connection to 127.12.250.126:42877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:51.508698 14214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.126:42877)
W20250903 08:12:51.508946 14214 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.250.126:42877)
W20250903 08:12:51.509105 14214 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.250.126:42877)
W20250903 08:12:51.509708 14214 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.250.126:42877)
I20250903 08:12:51.652977 14917 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:51.653213 14917 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:51.655151 14917 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:51.656230 14827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:51.657682 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:51.657905 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:51.657999 14827 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e; no voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:51.658331 14917 raft_consensus.cc:2747] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 08:12:52.391891 14954 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:52.392071 14954 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:52.392709 14954 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:52.394739 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:52.394894 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:52.394974 14827 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e; no voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:52.395222 14954 raft_consensus.cc:2747] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 08:12:53.390080 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 08:12:53.496932 14964 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:53.497139 14964 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:53.497862 14964 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:53.500236 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:12:53.500444 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:53.500560 14827 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba49201780c41ebb58a4c8c0ae7bb1e; no voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:53.500801 14964 raft_consensus.cc:2747] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250903 08:12:53.735504 14962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:53.735877 14962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:53.735996 14962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:53.744772 14962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:53.744918 14962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:53.745024 14962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:53.745090 14962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:53.755115 14962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:53.756886 14962 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:53.758839 14962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:53.767222 14972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:53.768445 14973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:53.768765 14975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:53.769042 14962 server_base.cc:1047] running on GCE node
I20250903 08:12:53.769951 14962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:53.771456 14962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:53.772679 14962 hybrid_clock.cc:648] HybridClock initialized: now 1756887173772617 us; error 62 us; skew 500 ppm
I20250903 08:12:53.773031 14962 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:53.775471 14962 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:53.776264 14962 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:53.776389 14962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:53.782143 14962 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250903 08:12:53.785321 14981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:53.786827 14962 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 08:12:53.787011 14962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:53.787793 14962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:53.818007 14962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:53.819041 14962 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:53.819329 14962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:53.840425 14962 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:53.840430 15032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:53.842267 14962 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:53.848834 15033 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:53.849603 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 14962
I20250903 08:12:53.852980 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:53.866809 15033 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:53.894975 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=25 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:12:53.895529 15033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:12:53.900843 15033 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:53.901607 15033 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:53.902220 15033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 4.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:53.903007 15037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:53.903290 15037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:53.904335 15033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:53.934439 15051 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:53.934655 15051 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:12:53.939803 15051 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:12:53.943504 15051 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:53.958962 14962 master_runner.cc:186] Error getting master registration for 127.12.250.124:39503: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111),
I20250903 08:12:53.962886 14962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:54.021608 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:54.021669 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:54.024484 14998 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:45739; Asking this server to re-register.
I20250903 08:12:54.024515 14997 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:36171; Asking this server to re-register.
I20250903 08:12:54.025297 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:54.025452 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:54.027240 14998 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:54.027379 14997 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:54.034291 15057 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:54.034529 15057 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:54.035228 15057 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:54.038180 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:54.043123 15008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 5 candidate_status { last_received { term: 4 index: 36 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
I20250903 08:12:54.043726 15008 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba49201780c41ebb58a4c8c0ae7bb1e in term 4.
I20250903 08:12:54.044301 14827 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters: ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:54.044628 15057 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250903 08:12:54.044775 15057 raft_consensus.cc:491] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:12:54.044893 15057 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:12:54.048004 15057 raft_consensus.cc:513] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:54.048878 15057 leader_election.cc:290] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 election: Requested vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:12:54.049464 15008 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" candidate_term: 5 candidate_status { last_received { term: 4 index: 36 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611"
I20250903 08:12:54.049702 15008 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 4 FOLLOWER]: Advancing to term 5
W20250903 08:12:54.052255 14827 leader_election.cc:336] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:54.054857 15008 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba49201780c41ebb58a4c8c0ae7bb1e in term 5.
I20250903 08:12:54.055537 14827 leader_election.cc:304] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e; no voters: ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:54.056010 15057 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 5 FOLLOWER]: Leader election won for term 5
I20250903 08:12:54.056530 15057 raft_consensus.cc:695] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [term 5 LEADER]: Becoming Leader. State: Replica: dba49201780c41ebb58a4c8c0ae7bb1e, State: Running, Role: LEADER
I20250903 08:12:54.057072 15057 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 36, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:54.059082 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba49201780c41ebb58a4c8c0ae7bb1e. Latest consensus state: current_term: 5 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:54.059422 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:54.060122 15063 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:12:54.062757 15063 catalog_manager.cc:671] Loaded metadata for table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:12:54.063215 15063 catalog_manager.cc:671] Loaded metadata for table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:12:54.063576 15063 catalog_manager.cc:671] Loaded metadata for table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]
I20250903 08:12:54.063942 15063 catalog_manager.cc:671] Loaded metadata for table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:12:54.064263 15063 catalog_manager.cc:671] Loaded metadata for table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:12:54.064613 15063 catalog_manager.cc:671] Loaded metadata for table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:12:54.064936 15063 catalog_manager.cc:671] Loaded metadata for table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:12:54.065304 15063 catalog_manager.cc:671] Loaded metadata for table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:12:54.065654 15063 catalog_manager.cc:671] Loaded metadata for table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:12:54.067565 15063 tablet_loader.cc:96] loaded metadata for tablet 0fc8d373f2f64e9c806440480e9e030d (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff])
I20250903 08:12:54.067902 15063 tablet_loader.cc:96] loaded metadata for tablet 4a67d22967884a02a6a89937065d2569 (table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995])
I20250903 08:12:54.068385 15063 tablet_loader.cc:96] loaded metadata for tablet 73649395ff784d5cb8ffff12c3e45dbe (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431])
I20250903 08:12:54.068665 15063 tablet_loader.cc:96] loaded metadata for tablet 879297b2be2c4717a747a95140139e6f (table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05])
I20250903 08:12:54.069097 15063 tablet_loader.cc:96] loaded metadata for tablet bdfe060848fc4e9d9534e539f39137da (table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b])
I20250903 08:12:54.069382 15063 tablet_loader.cc:96] loaded metadata for tablet bf91b0230f2742beac340a4c622f82ba (table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a])
I20250903 08:12:54.069674 15063 tablet_loader.cc:96] loaded metadata for tablet d1bd9a796f9947acb875272cf4e22c21 (table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff])
I20250903 08:12:54.069977 15063 tablet_loader.cc:96] loaded metadata for tablet da45d19c545a45c2abff9c05d8c0b293 (table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12])
I20250903 08:12:54.070443 15063 tablet_loader.cc:96] loaded metadata for tablet f65826010a9b482faf67a7120247eab1 (table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5])
I20250903 08:12:54.070684 15063 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:12:54.071719 15063 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:54.071802 15063 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:12:54.073074 15063 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:12:54.074014 15063 catalog_manager.cc:5966] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Loaded TSK: 0
I20250903 08:12:54.074874 15063 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:12:54.075757 15063 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:12:54.076606 15063 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:12:54.130813 14843 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49728:
name: "default.Table_6221cb658b3f49fb8d60a02df2b7a834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:54.131639 14837 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_feb51f00a18446ed842cbf54db2f3e06" } modify_external_catalogs: true
I20250903 08:12:54.131959 14837 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_feb51f00a18446ed842cbf54db2f3e06" } modify_external_catalogs: true
I20250903 08:12:54.132423 14838 master_service.cc:948] ReplaceTablet: received request to replace tablet 73649395ff784d5cb8ffff12c3e45dbe from {username='slave'} at 127.0.0.1:49728
I20250903 08:12:54.133083 14836 master_service.cc:948] ReplaceTablet: received request to replace tablet a43782057c894c77ad657f67deb11a71 from {username='slave'} at 127.0.0.1:49728
I20250903 08:12:54.135728 14839 master_service.cc:948] ReplaceTablet: received request to replace tablet 73649395ff784d5cb8ffff12c3e45dbe from {username='slave'} at 127.0.0.1:49728
I20250903 08:12:54.137463 14835 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49728:
name: "default.Table_0ecab4f09e824890b839f9ec45bfa3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:54.138830 14841 master_service.cc:948] ReplaceTablet: received request to replace tablet f65826010a9b482faf67a7120247eab1 from {username='slave'} at 127.0.0.1:49728
I20250903 08:12:54.139317 14842 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb" } modify_external_catalogs: true
I20250903 08:12:54.139370 14836 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_517de34d608c42cc9d4aeafea81f283f" } new_table_name: "default.Table_97201f0afe374761ad6c85589bf4ff69" modify_external_catalogs: true
I20250903 08:12:54.140813 14842 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_c04a578b55684bd69f07a226fc5dfdfb" } modify_external_catalogs: true
I20250903 08:12:54.139928 14840 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49728:
name: "default.Table_1a3f51ccc43c47e4abaeaa2384e5f332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:12:54.139060 14834 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49728:
name: "default.Table_94faf7a1e44247f6aeaed3759e0b4f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250903 08:12:54.142874 14897 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f2dfc21bdcd google::LogMessage::Fail() at ??:0
@ 0x7f2dfc21fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2dfc21b7cc google::LogMessage::Flush() at ??:0
@ 0x7f2dfc21ba11 google::LogMessage::~LogMessage() at ??:0
I20250903 08:12:54.155169 15008 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Refusing update from remote peer dba49201780c41ebb58a4c8c0ae7bb1e: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
W20250903 08:12:54.157416 14827 consensus_peers.cc:489] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e -> Peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Couldn't send request to peer ff9eecf53ccf4633bc701ff21e64ab0b. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:12:54.157842 15061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250903 08:12:54.166958 15037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba49201780c41ebb58a4c8c0ae7bb1e. Latest consensus state: current_term: 5 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:54.167224 15037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f2e1265e7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250903 08:12:54.169325 15064 mvcc.cc:204] Tried to move back new op lower bound from 7196209865325969408 to 7196209864942292992. Current Snapshot: MvccSnapshot[applied={T|T < 7196209865325969408}]
@ 0x7f2e1265d061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 08:12:54.173987 14838 catalog_manager.cc:5869] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Sending DeleteTablet for 3 replicas of tablet 73649395ff784d5cb8ffff12c3e45dbe
I20250903 08:12:54.176069 14838 catalog_manager.cc:6652] ReplaceTablet: tablet 73649395ff784d5cb8ffff12c3e45dbe deleted and replaced by tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f
@ 0x7f2e12664cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f2e12664b59 std::_Function_handler<>::_M_invoke() at ??:0
I20250903 08:12:54.193943 14838 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b" } new_table_name: "default.Table_05bff585c5694de286caf65b794fa796" modify_external_catalogs: true
I20250903 08:12:54.194015 14839 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_f7217c183b8a415d90880886722ccbb3" } new_table_name: "default.Table_3a84325d6c704721aaf5a796191718f2" modify_external_catalogs: true
I20250903 08:12:54.194460 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:54.194653 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e [sys.catalog]: This master's current role is: LEADER
I20250903 08:12:54.195256 14841 catalog_manager.cc:5869] T 00000000000000000000000000000000 P dba49201780c41ebb58a4c8c0ae7bb1e: Sending DeleteTablet for 3 replicas of tablet f65826010a9b482faf67a7120247eab1
I20250903 08:12:54.195842 14841 catalog_manager.cc:6652] ReplaceTablet: tablet f65826010a9b482faf67a7120247eab1 deleted and replaced by tablet a88988ebf6f04d55be358560c00ca932
I20250903 08:12:54.196192 14841 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49728:
table { table_name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c" } new_table_name: "default.Table_21c2b2a8c5c14a4b86cfe245eb1e81e9" modify_external_catalogs: true
I20250903 08:12:54.202165 15037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:54.202428 15037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:54.205819 13871 tablet_service.cc:1515] Processing DeleteTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe with delete_type TABLET_DATA_DELETED (replaced by tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f) from {username='slave'} at 127.0.0.1:38984
I20250903 08:12:54.205920 13869 tablet_service.cc:1515] Processing DeleteTablet for tablet f65826010a9b482faf67a7120247eab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a88988ebf6f04d55be358560c00ca932) from {username='slave'} at 127.0.0.1:38984
I20250903 08:12:54.212496 15079 tablet_replica.cc:331] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: stopping tablet replica
I20250903 08:12:54.212725 15079 raft_consensus.cc:2241] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:54.212868 14141 tablet_service.cc:1515] Processing DeleteTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe with delete_type TABLET_DATA_DELETED (replaced by tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f) from {username='slave'} at 127.0.0.1:52098
I20250903 08:12:54.212966 15079 raft_consensus.cc:2270] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:54.213032 14142 tablet_service.cc:1515] Processing DeleteTablet for tablet f65826010a9b482faf67a7120247eab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a88988ebf6f04d55be358560c00ca932) from {username='slave'} at 127.0.0.1:52098
I20250903 08:12:54.213716 15079 ts_tablet_manager.cc:1905] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:54.216243 14008 tablet_service.cc:1515] Processing DeleteTablet for tablet 73649395ff784d5cb8ffff12c3e45dbe with delete_type TABLET_DATA_DELETED (replaced by tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f) from {username='slave'} at 127.0.0.1:37214
I20250903 08:12:54.217105 14007 tablet_service.cc:1515] Processing DeleteTablet for tablet f65826010a9b482faf67a7120247eab1 with delete_type TABLET_DATA_DELETED (replaced by tablet a88988ebf6f04d55be358560c00ca932) from {username='slave'} at 127.0.0.1:37214
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 08:12:54.228001 15081 tablet_replica.cc:331] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: stopping tablet replica
I20250903 08:12:54.228210 15081 raft_consensus.cc:2241] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:12:54.228926 15081 raft_consensus.cc:2270] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f2dfef842b9 kudu::Thread::SuperviseThread() at ??:0
I20250903 08:12:54.229609 15081 ts_tablet_manager.cc:1905] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:54.230322 15079 ts_tablet_manager.cc:1918] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.230456 15079 log.cc:1199] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/wals/73649395ff784d5cb8ffff12c3e45dbe
I20250903 08:12:54.230927 15079 ts_tablet_manager.cc:1939] T 73649395ff784d5cb8ffff12c3e45dbe P 70c467196f6846b78b5a7c7ac8a2787d: Deleting consensus metadata
I20250903 08:12:54.232085 15079 tablet_replica.cc:331] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: stopping tablet replica
@ 0x7f2e023a86db start_thread at ??:0
I20250903 08:12:54.232323 15079 raft_consensus.cc:2241] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:12:54.232292 14830 catalog_manager.cc:4928] TS 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323): tablet 73649395ff784d5cb8ffff12c3e45dbe (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]) successfully deleted
I20250903 08:12:54.232867 15079 raft_consensus.cc:2270] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:54.233517 15079 ts_tablet_manager.cc:1905] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f2dfb0cc71f clone at ??:0
I20250903 08:12:54.239850 15081 ts_tablet_manager.cc:1918] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.239982 15081 log.cc:1199] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/wals/73649395ff784d5cb8ffff12c3e45dbe
I20250903 08:12:54.240260 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:54.240437 15081 ts_tablet_manager.cc:1939] T 73649395ff784d5cb8ffff12c3e45dbe P 668be13bcd854aa8afa97d2bbaf067a7: Deleting consensus metadata
I20250903 08:12:54.240598 15080 tablet_replica.cc:331] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: stopping tablet replica
I20250903 08:12:54.240806 15080 raft_consensus.cc:2241] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:54.241025 15079 ts_tablet_manager.cc:1918] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.241176 15079 log.cc:1199] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/wals/f65826010a9b482faf67a7120247eab1
I20250903 08:12:54.242039 14997 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:54555; Asking this server to re-register.
I20250903 08:12:54.242688 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:54.242700 15079 ts_tablet_manager.cc:1939] T f65826010a9b482faf67a7120247eab1 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting consensus metadata
I20250903 08:12:54.241046 15080 raft_consensus.cc:2270] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:54.243877 15081 tablet_replica.cc:331] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: stopping tablet replica
I20250903 08:12:54.244096 15081 raft_consensus.cc:2241] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:54.244284 15081 raft_consensus.cc:2270] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:54.244573 14997 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:54.244935 15081 ts_tablet_manager.cc:1905] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:54.245702 15080 ts_tablet_manager.cc:1905] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:54.251699 15081 ts_tablet_manager.cc:1918] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.251847 15081 log.cc:1199] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/wals/f65826010a9b482faf67a7120247eab1
I20250903 08:12:54.252341 15081 ts_tablet_manager.cc:1939] T f65826010a9b482faf67a7120247eab1 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting consensus metadata
I20250903 08:12:54.252646 15080 ts_tablet_manager.cc:1918] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.252770 15080 log.cc:1199] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/wals/73649395ff784d5cb8ffff12c3e45dbe
I20250903 08:12:54.253217 15080 ts_tablet_manager.cc:1939] T 73649395ff784d5cb8ffff12c3e45dbe P 0b68b07307444fdebbee0bff3fb79869: Deleting consensus metadata
I20250903 08:12:54.254894 15080 tablet_replica.cc:331] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: stopping tablet replica
I20250903 08:12:54.255110 15080 raft_consensus.cc:2241] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:12:54.255304 15080 raft_consensus.cc:2270] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:12:54.255890 15080 ts_tablet_manager.cc:1905] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:12:54.262252 15080 ts_tablet_manager.cc:1918] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:12:54.262431 15080 log.cc:1199] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/wals/f65826010a9b482faf67a7120247eab1
I20250903 08:12:54.262845 15080 ts_tablet_manager.cc:1939] T f65826010a9b482faf67a7120247eab1 P 0b68b07307444fdebbee0bff3fb79869: Deleting consensus metadata
W20250903 08:12:54.426335 14096 connection.cc:537] server connection from 127.0.0.1:52098 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.426321 13823 connection.cc:537] server connection from 127.0.0.1:38984 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.426752 14982 connection.cc:537] server connection from 127.0.0.1:44726 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.427321 13962 connection.cc:537] server connection from 127.0.0.1:37214 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.427629 14215 connection.cc:537] client connection to 127.12.250.125:34047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.427796 13825 connection.cc:537] client connection to 127.12.250.125:34047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:12:54.428787 14215 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.250.125:34047)
W20250903 08:12:54.429468 14215 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.125:34047)
W20250903 08:12:54.429805 14215 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.250.125:34047)
W20250903 08:12:54.430552 14215 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.250.125:34047)
I20250903 08:12:54.736503 15091 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dba49201780c41ebb58a4c8c0ae7bb1e)
I20250903 08:12:54.736701 15091 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:54.738562 15091 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:54.739673 14983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:54.741142 14982 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:54.741381 14983 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:54.741559 14983 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:54.741905 15091 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:12:55.246728 15097 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dba49201780c41ebb58a4c8c0ae7bb1e)
I20250903 08:12:55.246909 15097 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:55.247613 15097 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:55.250186 14982 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:55.250380 14983 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:55.250500 14983 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:55.250706 15097 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:12:56.009218 13289 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu pid 14962
I20250903 08:12:56.016001 15028 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 08:12:56.123696 13289 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu with pid 14962
I20250903 08:12:56.873672 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:12:57.198485 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:12:57.198843 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:12:57.198963 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:12:57.207345 15107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:12:57.207456 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:12:57.207533 15107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:12:57.207595 15107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:12:57.217842 15107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:12:57.219560 15107 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:12:57.221526 15107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:12:57.230224 15113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:57.230224 15115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:12:57.230255 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:12:57.231685 15107 server_base.cc:1047] running on GCE node
I20250903 08:12:57.232612 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:12:57.234088 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:12:57.235299 15107 hybrid_clock.cc:648] HybridClock initialized: now 1756887177235250 us; error 44 us; skew 500 ppm
I20250903 08:12:57.235643 15107 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:12:57.237849 15107 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:12:57.238590 15107 fs_manager.cc:362] Metadata directory not provided
I20250903 08:12:57.238699 15107 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:12:57.244617 15107 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 08:12:57.247815 15121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:12:57.249559 15107 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250903 08:12:57.249748 15107 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:12:57.250522 15107 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:12:57.281746 15107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:12:57.282663 15107 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:12:57.282918 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:12:57.302961 15107 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:12:57.302963 15172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:12:57.305608 15107 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:12:57.313004 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 15107
I20250903 08:12:57.313621 15174 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:12:57.319252 15174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:12:57.332649 15174 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:12:57.362715 15174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=27 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:12:57.363289 15174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:12:57.368570 15174 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:57.369309 15174 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:12:57.369966 15174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 5.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:57.370734 15178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:12:57.371007 15178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:12:57.371778 15174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:12:57.385635 15192 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:12:57.385851 15192 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:12:57.390949 15192 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:12:57.394243 15192 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:12:57.394724 15107 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:12:57.400197 15107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:12:57.634891 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:57.635722 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:57.637770 15137 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:57185; Asking this server to re-register.
I20250903 08:12:57.637797 15138 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:59243; Asking this server to re-register.
I20250903 08:12:57.638628 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:57.638710 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:57.640512 15137 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:12:57.640619 15138 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:12:57.761870 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:12:57.763068 15138 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:50965; Asking this server to re-register.
I20250903 08:12:57.763602 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:12:57.764699 15138 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:12:57.886986 15197 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:57.887262 15197 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:57.889006 15197 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:57.889577 15123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:57.890854 15123 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:12:57.891085 15122 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:12:57.891201 15122 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:57.891531 15197 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:12:58.484898 15200 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:58.485080 15200 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:58.485713 15200 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:58.487696 15122 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:58.487936 15123 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:58.488085 15123 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:58.488368 15200 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:12:59.210954 15206 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:12:59.211144 15206 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:12:59.211743 15206 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:12:59.216014 15122 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:12:59.216226 15123 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:12:59.216327 15123 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:12:59.216535 15206 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:12:59.828022 13289 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu pid 15107
I20250903 08:12:59.840399 15168 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 08:12:59.949740 13289 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu with pid 15107
I20250903 08:13:00.532634 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:13:00.868561 15214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:13:00.868933 15214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:13:00.869019 15214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:13:00.877455 15214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:13:00.877610 15214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:13:00.877712 15214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:13:00.877796 15214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:13:00.888013 15214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:13:00.889878 15214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:13:00.891692 15214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:13:00.899658 15221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:00.899798 15220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:00.900496 15223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:13:00.901305 15214 server_base.cc:1047] running on GCE node
I20250903 08:13:00.902096 15214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:13:00.903563 15214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:13:00.904796 15214 hybrid_clock.cc:648] HybridClock initialized: now 1756887180904737 us; error 47 us; skew 500 ppm
I20250903 08:13:00.905176 15214 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:13:00.907413 15214 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:13:00.908166 15214 fs_manager.cc:362] Metadata directory not provided
I20250903 08:13:00.908274 15214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:13:00.913796 15214 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 08:13:00.916990 15229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:13:00.918538 15214 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 08:13:00.918730 15214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:13:00.919471 15214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:13:00.941057 15214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:13:00.942107 15214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:13:00.942502 15214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:13:00.962502 15214 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:13:00.962543 15280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:13:00.964196 15214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:13:00.970963 15281 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:13:00.972514 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 15214
I20250903 08:13:00.975224 15281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:13:00.988528 15281 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:13:01.020148 15281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=27 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:13:01.020785 15281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:13:01.026329 15281 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:01.027055 15281 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:13:01.027767 15281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 5.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:01.028541 15285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:01.028808 15285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:01.029661 15281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:13:01.053627 15214 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:01.059399 15214 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:01.060319 15299 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:13:01.060475 15299 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:13:01.065012 15299 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:13:01.068526 15299 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:13:01.161005 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:01.161157 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:01.163686 15246 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:34723; Asking this server to re-register.
I20250903 08:13:01.163712 15245 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:59055; Asking this server to re-register.
I20250903 08:13:01.164458 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:01.164610 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:01.166374 15246 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:13:01.166530 15245 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:01.284009 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:01.285815 15245 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:54375; Asking this server to re-register.
I20250903 08:13:01.286569 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:01.288074 15245 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:13:01.518296 15285 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:01.518641 15285 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:01.520632 15285 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:01.521570 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:01.523341 15230 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:01.523535 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:01.523707 15231 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:01.524051 15285 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:13:02.165848 15308 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:02.166052 15308 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:02.166859 15308 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:02.168849 15230 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:02.169112 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:02.169229 15231 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:02.169576 15308 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:13:02.979698 15313 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:02.979918 15313 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:02.980567 15313 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:02.982784 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:02.982978 15230 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:02.983125 15230 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:02.983378 15313 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:13:03.796983 15319 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:03.797178 15319 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:03.797991 15319 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:03.800156 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:03.800341 15230 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:03.800503 15230 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:03.800770 15319 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:13:03.998713 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.124:39503
--webserver_interface=127.12.250.124
--webserver_port=32903
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:13:04.339677 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:13:04.340029 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:13:04.340109 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:13:04.348249 15324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:13:04.348356 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:13:04.348412 15324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:13:04.348459 15324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:13:04.358568 15324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.124:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.250.124
--webserver_port=32903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:13:04.360200 15324 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:13:04.362022 15324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:13:04.370489 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:04.370584 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:04.370540 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:13:04.371620 15324 server_base.cc:1047] running on GCE node
I20250903 08:13:04.372447 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:13:04.374116 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:13:04.375357 15324 hybrid_clock.cc:648] HybridClock initialized: now 1756887184375280 us; error 56 us; skew 500 ppm
I20250903 08:13:04.375800 15324 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:13:04.378263 15324 webserver.cc:480] Webserver started at http://127.12.250.124:32903/ using document root <none> and password file <none>
I20250903 08:13:04.379004 15324 fs_manager.cc:362] Metadata directory not provided
I20250903 08:13:04.379108 15324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:13:04.384493 15324 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 08:13:04.387532 15338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:13:04.389108 15324 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:13:04.389298 15324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
format_stamp: "Formatted at 2025-09-03 08:12:34 on dist-test-slave-knz7"
I20250903 08:13:04.390024 15324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:13:04.419697 15324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:13:04.420610 15324 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:13:04.420889 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:13:04.442390 15324 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.124:39503
I20250903 08:13:04.442467 15389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.124:39503 every 8 connection(s)
I20250903 08:13:04.444131 15324 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-2/data/info.pb
I20250903 08:13:04.449424 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 15324
I20250903 08:13:04.453199 15390 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:13:04.458103 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Bootstrap starting.
I20250903 08:13:04.470531 15390 log.cc:826] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Log is configured to *not* fsync() on all Append() calls
I20250903 08:13:04.490667 14207 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:13:04.501111 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250903 08:13:04.501780 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Bootstrap complete.
I20250903 08:13:04.507632 15390 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:04.510658 15390 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9eecf53ccf4633bc701ff21e64ab0b, State: Initialized, Role: FOLLOWER
I20250903 08:13:04.511363 15390 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:04.512130 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:04.512388 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:04.513232 15390 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:13:04.528589 15409 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:13:04.529000 15409 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: loading cluster ID for follower catalog manager: success
I20250903 08:13:04.533897 15409 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: acquiring CA information for follower catalog manager: success
I20250903 08:13:04.537078 15409 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:13:04.546653 15324 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:04.550004 15324 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:04.731009 14073 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:13:04.733500 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:55643; Asking this server to re-register.
I20250903 08:13:04.734228 14073 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:04.735883 15355 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:13:04.829524 13936 heartbeater.cc:344] Connected to a master server at 127.12.250.124:39503
I20250903 08:13:04.831856 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:41277; Asking this server to re-register.
I20250903 08:13:04.832640 13936 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:04.834173 15355 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:13:04.997031 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:44951; Asking this server to re-register.
I20250903 08:13:04.997831 14207 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:04.999177 15355 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:05.110446 15415 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:05.110718 15415 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:05.112550 15415 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
I20250903 08:13:05.113359 15256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
W20250903 08:13:05.113517 15340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:05.113905 15256 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9eecf53ccf4633bc701ff21e64ab0b for earlier term 4. Current term is 5.
I20250903 08:13:05.114773 15339 leader_election.cc:400] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9eecf53ccf4633bc701ff21e64ab0b for earlier term 4. Current term is 5.
I20250903 08:13:05.114856 15339 leader_election.cc:403] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20250903 08:13:05.115008 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:05.115147 15415 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 3 FOLLOWER]: Advancing to term 5
I20250903 08:13:05.118500 15415 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9eecf53ccf4633bc701ff21e64ab0b for earlier term 4. Current term is 5.
I20250903 08:13:05.715485 15422 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:05.715663 15422 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:05.716406 15422 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
I20250903 08:13:05.716948 15256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 6 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
I20250903 08:13:05.717234 15256 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff9eecf53ccf4633bc701ff21e64ab0b for term 6 because replica has last-logged OpId of term: 5 index: 40, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 35.
W20250903 08:13:05.718469 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:05.718596 15340 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9eecf53ccf4633bc701ff21e64ab0b; no voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:05.718946 15422 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 08:13:05.848878 15425 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:05.849128 15425 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:05.849917 15425 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:05.852231 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:05.856835 15365 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 6 candidate_status { last_received { term: 5 index: 40 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" is_pre_election: true
I20250903 08:13:05.857234 15365 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 5.
I20250903 08:13:05.857777 15230 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:05.858110 15425 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250903 08:13:05.858237 15425 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:05.858358 15425 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:13:05.862192 15425 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:05.862882 15425 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 election: Requested vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
I20250903 08:13:05.863530 15365 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e5331161d9e4df39004183d79dc3611" candidate_term: 6 candidate_status { last_received { term: 5 index: 40 } } ignore_live_leader: false dest_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b"
I20250903 08:13:05.863762 15365 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 5 FOLLOWER]: Advancing to term 6
W20250903 08:13:05.864365 15231 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:05.866065 15365 raft_consensus.cc:2466] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5331161d9e4df39004183d79dc3611 in term 6.
I20250903 08:13:05.866560 15230 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:05.866847 15425 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Leader election won for term 6
I20250903 08:13:05.867291 15425 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 LEADER]: Becoming Leader. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Running, Role: LEADER
I20250903 08:13:05.867776 15425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 5.40, Last appended by leader: 40, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:05.870494 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:05.870832 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: LEADER
I20250903 08:13:05.871522 15432 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:13:05.874121 15432 catalog_manager.cc:671] Loaded metadata for table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:13:05.874560 15432 catalog_manager.cc:671] Loaded metadata for table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:13:05.874892 15432 catalog_manager.cc:671] Loaded metadata for table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]
I20250903 08:13:05.875218 15432 catalog_manager.cc:671] Loaded metadata for table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:13:05.875504 15432 catalog_manager.cc:671] Loaded metadata for table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:13:05.875771 15432 catalog_manager.cc:671] Loaded metadata for table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:13:05.876037 15432 catalog_manager.cc:671] Loaded metadata for table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:13:05.876322 15432 catalog_manager.cc:671] Loaded metadata for table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:13:05.876593 15432 catalog_manager.cc:671] Loaded metadata for table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:13:05.878556 15432 tablet_loader.cc:96] loaded metadata for tablet 0fc8d373f2f64e9c806440480e9e030d (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff])
I20250903 08:13:05.878875 15432 tablet_loader.cc:96] loaded metadata for tablet 4a67d22967884a02a6a89937065d2569 (table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995])
I20250903 08:13:05.879120 15432 tablet_loader.cc:96] loaded metadata for tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431])
I20250903 08:13:05.879611 15432 tablet_loader.cc:96] loaded metadata for tablet 879297b2be2c4717a747a95140139e6f (table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05])
I20250903 08:13:05.879964 15432 tablet_loader.cc:96] loaded metadata for tablet a88988ebf6f04d55be358560c00ca932 (table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5])
I20250903 08:13:05.880175 15432 tablet_loader.cc:96] loaded metadata for tablet bdfe060848fc4e9d9534e539f39137da (table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b])
I20250903 08:13:05.880416 15432 tablet_loader.cc:96] loaded metadata for tablet bf91b0230f2742beac340a4c622f82ba (table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a])
I20250903 08:13:05.880658 15432 tablet_loader.cc:96] loaded metadata for tablet d1bd9a796f9947acb875272cf4e22c21 (table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff])
I20250903 08:13:05.880890 15432 tablet_loader.cc:96] loaded metadata for tablet da45d19c545a45c2abff9c05d8c0b293 (table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12])
I20250903 08:13:05.881454 15432 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:13:05.882467 15432 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:13:05.882543 15432 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:13:05.883700 15432 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:13:05.884539 15432 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Loaded TSK: 0
I20250903 08:13:05.885493 15432 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:13:05.886345 15432 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:13:05.887154 15432 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:13:05.997586 15245 master_service.cc:948] ReplaceTablet: received request to replace tablet bdfe060848fc4e9d9534e539f39137da from {username='slave'} at 127.0.0.1:37112
I20250903 08:13:05.997725 15246 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a67d22967884a02a6a89937065d2569 from {username='slave'} at 127.0.0.1:37112
I20250903 08:13:05.998284 15243 master_service.cc:948] ReplaceTablet: received request to replace tablet 879297b2be2c4717a747a95140139e6f from {username='slave'} at 127.0.0.1:37112
I20250903 08:13:05.998525 15244 master_service.cc:948] ReplaceTablet: received request to replace tablet da45d19c545a45c2abff9c05d8c0b293 from {username='slave'} at 127.0.0.1:37112
I20250903 08:13:05.998531 15242 master_service.cc:948] ReplaceTablet: received request to replace tablet 09070dc67d994d26a5c768db60faf96c from {username='slave'} at 127.0.0.1:37112
I20250903 08:13:05.999583 15239 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37112:
name: "default.Table_c88874dd846641f19e00bc8d13e3377b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:05.999636 15241 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37112:
name: "default.Table_3b5331a45947402b9deafb9d63b55b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:06.001358 15237 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37112:
table { table_name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58" } modify_external_catalogs: true
I20250903 08:13:06.001629 15237 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37112:
table { table_name: "default.Table_cfed4bd13e9b4b9f843905f4874d6f58" } modify_external_catalogs: true
I20250903 08:13:06.002764 15240 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37112:
name: "default.Table_58f55395478b4968a5f5e04b0139ac01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:06.005618 15238 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37112:
table { table_name: "default.Table_fc56d0f82c054a9facf698e29514b8d4" } modify_external_catalogs: true
I20250903 08:13:06.006454 15238 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37112:
table { table_name: "default.Table_fc56d0f82c054a9facf698e29514b8d4" } modify_external_catalogs: true
F20250903 08:13:06.008766 15298 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f1434486dcd google::LogMessage::Fail() at ??:0
@ 0x7f143448ab93 google::LogMessage::SendToLog() at ??:0
I20250903 08:13:06.014111 15365 raft_consensus.cc:1273] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Refusing update from remote peer 1e5331161d9e4df39004183d79dc3611: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 6 index: 42. (index mismatch)
I20250903 08:13:06.008224 15242 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37112:
name: "default.Table_6a11e40b85c9416abb127a1c1ecf1e0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:13:06.015120 15231 consensus_peers.cc:489] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 -> Peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Couldn't send request to peer dba49201780c41ebb58a4c8c0ae7bb1e. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:13:06.015435 15430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 32, Time since last communication: 0.000s
@ 0x7f14344867cc google::LogMessage::Flush() at ??:0
@ 0x7f1434486a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f144a8c97b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f144a8c8061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 08:13:06.035588 15438 mvcc.cc:204] Tried to move back new op lower bound from 7196209865325969408 to 7196209850750582784. Current Snapshot: MvccSnapshot[applied={T|T < 7196209865325969408 or (T in {7196209865325969408})}]
@ 0x7f144a8cfcdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250903 08:13:06.037218 15440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:06.037618 15440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:06.037667 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:06.037883 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:06.038092 15422 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e5331161d9e4df39004183d79dc3611. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:06.038206 15422 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:06.038404 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:06.038552 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f144a8cfb59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 08:13:06.046141 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "1e5331161d9e4df39004183d79dc3611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:06.046429 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f14371ef2b9 kudu::Thread::SuperviseThread() at ??:0
I20250903 08:13:06.050329 15243 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Sending DeleteTablet for 3 replicas of tablet 879297b2be2c4717a747a95140139e6f
@ 0x7f143a6136db start_thread at ??:0
I20250903 08:13:06.052699 15243 catalog_manager.cc:6652] ReplaceTablet: tablet 879297b2be2c4717a747a95140139e6f deleted and replaced by tablet 40c1fd973fad416eb787f5340b5352ce
@ 0x7f143333771f clone at ??:0
W20250903 08:13:06.234462 14095 connection.cc:537] client connection to 127.12.250.126:42877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:06.234797 14214 connection.cc:537] client connection to 127.12.250.126:42877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:06.234966 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:13:06.052681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.12.250.66:42745)
0903 08:13:06.053688 (+ 1007us) server_negotiation.cc:207] Beginning negotiation
0903 08:13:06.053698 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 08:13:06.053724 (+ 26us) server_negotiation.cc:408] Connection header received
0903 08:13:06.053809 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:13:06.053817 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:13:06.053904 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:13:06.054025 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:13:06.234150 (+180125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48612: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":768,"threads_started":1}
W20250903 08:13:06.235419 13823 connection.cc:537] client connection to 127.12.250.126:42877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:06.235471 14214 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.250.126:42877)
W20250903 08:13:06.235700 14214 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.250.126:42877)
W20250903 08:13:06.235931 14214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.126:42877)
W20250903 08:13:06.237157 14214 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.250.126:42877)
W20250903 08:13:06.237612 15454 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 08:13:06.053890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.12.250.65:45323)
0903 08:13:06.055059 (+ 1169us) server_negotiation.cc:207] Beginning negotiation
0903 08:13:06.055068 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0903 08:13:06.055090 (+ 22us) server_negotiation.cc:408] Connection header received
0903 08:13:06.055181 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 08:13:06.055189 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 08:13:06.055277 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 08:13:06.055392 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 08:13:06.234166 (+178774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34496: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":896,"threads_started":1}
I20250903 08:13:06.541013 15445 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:13:06.541321 15445 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:06.542214 15445 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
W20250903 08:13:06.544582 15339 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:13:06.544906 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:06.545068 15340 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9eecf53ccf4633bc701ff21e64ab0b; no voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:06.545365 15445 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 08:13:07.236255 15468 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:13:07.236449 15468 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:07.237172 15468 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
W20250903 08:13:07.239337 15339 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877): Network error: Client connection negotiation failed: client connection to 127.12.250.126:42877: connect: Connection refused (error 111)
W20250903 08:13:07.239524 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:07.239629 15340 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff9eecf53ccf4633bc701ff21e64ab0b; no voters: 1e5331161d9e4df39004183d79dc3611, dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:07.239884 15468 raft_consensus.cc:2747] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 08:13:07.349581 13289 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
/tmp/dist-test-taskVDqtkV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--webserver_interface=127.12.250.126
--webserver_port=41743
--builtin_ntp_servers=127.12.250.84:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 08:13:07.696658 15473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 08:13:07.696993 15473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 08:13:07.697090 15473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 08:13:07.705040 15473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 08:13:07.705153 15473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 08:13:07.705209 15473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 08:13:07.705286 15473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 08:13:07.715284 15473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.250.84:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37817
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.250.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.250.126
--webserver_port=41743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 07:50:26 UTC on 5fd53c4cbb9d
build id 7876
ASAN enabled
I20250903 08:13:07.717082 15473 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:13:07.719259 15473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:13:07.728484 15479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:07.728415 15478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:13:07.728487 15481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:13:07.729703 15473 server_base.cc:1047] running on GCE node
I20250903 08:13:07.730520 15473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 08:13:07.732098 15473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 08:13:07.733356 15473 hybrid_clock.cc:648] HybridClock initialized: now 1756887187733310 us; error 43 us; skew 500 ppm
I20250903 08:13:07.733723 15473 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 08:13:07.736147 15473 webserver.cc:480] Webserver started at http://127.12.250.126:41743/ using document root <none> and password file <none>
I20250903 08:13:07.736940 15473 fs_manager.cc:362] Metadata directory not provided
I20250903 08:13:07.737058 15473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:13:07.743469 15473 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250903 08:13:07.746582 15487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:13:07.748085 15473 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 08:13:07.748277 15473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data,/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
uuid: "1e5331161d9e4df39004183d79dc3611"
format_stamp: "Formatted at 2025-09-03 08:12:33 on dist-test-slave-knz7"
I20250903 08:13:07.749107 15473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:13:07.762547 15473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:13:07.763386 15473 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:13:07.763715 15473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:13:07.784085 15473 rpc_server.cc:307] RPC server started. Bound to: 127.12.250.126:42877
I20250903 08:13:07.784103 15538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.250.126:42877 every 8 connection(s)
I20250903 08:13:07.785956 15473 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/master-0/data/info.pb
I20250903 08:13:07.792541 15539 sys_catalog.cc:263] Verifying existing consensus state
I20250903 08:13:07.795706 13289 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVDqtkV/build/asan/bin/kudu as pid 15473
I20250903 08:13:07.796175 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap starting.
I20250903 08:13:07.809172 15539 log.cc:826] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Log is configured to *not* fsync() on all Append() calls
I20250903 08:13:07.819957 14074 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:07.847105 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=43 ignored=0} inserts{seen=29 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250903 08:13:07.847810 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: Bootstrap complete.
I20250903 08:13:07.853070 15539 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:07.856390 15539 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5331161d9e4df39004183d79dc3611, State: Initialized, Role: FOLLOWER
I20250903 08:13:07.857170 15539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 6.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:07.857985 15544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:07.858254 15544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 08:13:07.859042 15539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:13:07.876600 15558 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:13:07.876847 15558 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: loading cluster ID for follower catalog manager: success
I20250903 08:13:07.881807 15558 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: acquiring CA information for follower catalog manager: success
I20250903 08:13:07.886641 15558 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 08:13:07.883332 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:07.890662 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:08.009790 15562 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:13:08.010058 15562 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:08.010766 15562 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
W20250903 08:13:08.013454 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:08.018414 15514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 7 candidate_status { last_received { term: 6 index: 45 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611" is_pre_election: true
I20250903 08:13:08.018963 15514 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff9eecf53ccf4633bc701ff21e64ab0b in term 6.
I20250903 08:13:08.019510 15339 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:08.020012 15562 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250903 08:13:08.020167 15562 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1e5331161d9e4df39004183d79dc3611)
I20250903 08:13:08.020256 15562 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:13:08.023291 15562 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:08.024068 15562 leader_election.cc:290] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 election: Requested vote from peers 1e5331161d9e4df39004183d79dc3611 (127.12.250.126:42877), dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047)
I20250903 08:13:08.024533 15514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" candidate_term: 7 candidate_status { last_received { term: 6 index: 45 } } ignore_live_leader: false dest_uuid: "1e5331161d9e4df39004183d79dc3611"
I20250903 08:13:08.024787 15514 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 6 FOLLOWER]: Advancing to term 7
W20250903 08:13:08.025611 15340 leader_election.cc:336] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:08.028731 15514 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff9eecf53ccf4633bc701ff21e64ab0b in term 7.
I20250903 08:13:08.029227 15339 leader_election.cc:304] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611, ff9eecf53ccf4633bc701ff21e64ab0b; no voters: dba49201780c41ebb58a4c8c0ae7bb1e
I20250903 08:13:08.029577 15562 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 7 FOLLOWER]: Leader election won for term 7
I20250903 08:13:08.030017 15562 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [term 7 LEADER]: Becoming Leader. State: Replica: ff9eecf53ccf4633bc701ff21e64ab0b, State: Running, Role: LEADER
I20250903 08:13:08.030565 15562 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 6.45, Last appended by leader: 45, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:08.032487 15566 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9eecf53ccf4633bc701ff21e64ab0b. Latest consensus state: current_term: 7 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:08.032770 15566 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: LEADER
I20250903 08:13:08.033514 15568 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:13:08.036096 15568 catalog_manager.cc:671] Loaded metadata for table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]
I20250903 08:13:08.036557 15568 catalog_manager.cc:671] Loaded metadata for table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]
I20250903 08:13:08.036916 15568 catalog_manager.cc:671] Loaded metadata for table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff]
I20250903 08:13:08.037329 15568 catalog_manager.cc:671] Loaded metadata for table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05]
I20250903 08:13:08.037761 15568 catalog_manager.cc:671] Loaded metadata for table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]
I20250903 08:13:08.038192 15568 catalog_manager.cc:671] Loaded metadata for table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431]
I20250903 08:13:08.038566 15568 catalog_manager.cc:671] Loaded metadata for table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]
I20250903 08:13:08.038916 15568 catalog_manager.cc:671] Loaded metadata for table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]
I20250903 08:13:08.039322 15568 catalog_manager.cc:671] Loaded metadata for table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a]
I20250903 08:13:08.041476 15568 tablet_loader.cc:96] loaded metadata for tablet 0fc8d373f2f64e9c806440480e9e030d (table default.table_ab9942373e884c1490e9c1fb31e1b23b [id=53236081245549cdbf54ad2a3c947bff])
I20250903 08:13:08.041771 15568 tablet_loader.cc:96] loaded metadata for tablet 40c1fd973fad416eb787f5340b5352ce (table default.table_cfed4bd13e9b4b9f843905f4874d6f58 [id=601627dee6be420ea59bd2df2ffc0d05])
I20250903 08:13:08.042339 15568 tablet_loader.cc:96] loaded metadata for tablet 4a8ccb65e0f04c39b3b7fc0b932ceb6f (table default.table_f7217c183b8a415d90880886722ccbb3 [id=898f43ef947b4536bd925aa16a528431])
I20250903 08:13:08.043257 15568 tablet_loader.cc:96] loaded metadata for tablet a578aed41b9d4b119932d6c4b4ea80de (table default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b])
I20250903 08:13:08.043514 15568 tablet_loader.cc:96] loaded metadata for tablet a88988ebf6f04d55be358560c00ca932 (table default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5])
I20250903 08:13:08.043962 15568 tablet_loader.cc:96] loaded metadata for tablet bf91b0230f2742beac340a4c622f82ba (table default.table_f9ba5ac45a0c4a19bbb7be1548642a9c [id=f271a3d5643d4a6da2bcfdd00386241a])
I20250903 08:13:08.044216 15568 tablet_loader.cc:96] loaded metadata for tablet cf5e4ade871e4e8fba678ffe244b2f83 (table default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995])
I20250903 08:13:08.044498 15568 tablet_loader.cc:96] loaded metadata for tablet d1bd9a796f9947acb875272cf4e22c21 (table default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff])
I20250903 08:13:08.045106 15568 tablet_loader.cc:96] loaded metadata for tablet f321c10ca8ce4917b3ccb2ef22d80bca (table default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12])
I20250903 08:13:08.045569 15568 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:13:08.046650 15568 catalog_manager.cc:1261] Loaded cluster ID: fbe02324d9d04448be7fb82999004b09
I20250903 08:13:08.046725 15568 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:13:08.048028 15568 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:13:08.049167 15568 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Loaded TSK: 0
I20250903 08:13:08.050077 15568 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:13:08.051054 15568 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250903 08:13:08.051941 15568 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250903 08:13:08.075585 15354 master_service.cc:948] ReplaceTablet: received request to replace tablet d1bd9a796f9947acb875272cf4e22c21 from {username='slave'} at 127.0.0.1:38696
I20250903 08:13:08.076851 15347 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38696:
name: "default.Table_0ecab4f09e824890b839f9ec45bfa3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:08.077620 15346 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38696:
name: "default.Table_3b5331a45947402b9deafb9d63b55b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:08.077925 15348 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38696:
name: "default.Table_94faf7a1e44247f6aeaed3759e0b4f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:08.079028 15353 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38696:
name: "default.Table_250ab81379d64db694b3b9013da7f8a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 08:13:08.081534 15349 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38696:
table { table_name: "default.Table_517de34d608c42cc9d4aeafea81f283f" } new_table_name: "default.Table_97201f0afe374761ad6c85589bf4ff69" modify_external_catalogs: true
I20250903 08:13:08.081699 15355 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38696:
table { table_name: "default.Table_b21f9eea82374d75a52c4b6787afdf48" } new_table_name: "default.Table_bde72d06420747c19ba769641487520e" modify_external_catalogs: true
I20250903 08:13:08.081668 15351 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38696:
table { table_name: "default.Table_f7217c183b8a415d90880886722ccbb3" } new_table_name: "default.Table_3a84325d6c704721aaf5a796191718f2" modify_external_catalogs: true
I20250903 08:13:08.082444 15350 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38696:
table { table_name: "default.Table_ab9942373e884c1490e9c1fb31e1b23b" } new_table_name: "default.Table_05bff585c5694de286caf65b794fa796" modify_external_catalogs: true
I20250903 08:13:08.082846 15352 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38696:
table { table_name: "default.Table_f9ba5ac45a0c4a19bbb7be1548642a9c" } new_table_name: "default.Table_21c2b2a8c5c14a4b86cfe245eb1e81e9" modify_external_catalogs: true
I20250903 08:13:08.090293 15514 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Refusing update from remote peer ff9eecf53ccf4633bc701ff21e64ab0b: Log matching property violated. Preceding OpId in replica: term: 6 index: 45. Preceding OpId from leader: term: 7 index: 47. (index mismatch)
F20250903 08:13:08.090606 15408 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_3f66d15db9bc4be7bf75984378182586, message={"server":"","servicePrincipal":"","db":"default","table":"table_3f66d15db9bc4be7bf75984378182586","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_3f66d15db9bc4be7bf75984378182586\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_3f66d15db9bc4be7bf75984378182586\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_3f66d15db9bc4be7bf75984378182586\",\"kudu.table_id\":\"2f089ec4af564830b0beba7468c53d08\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1756887157, eventType=CREATE_TABLE, dbName=default, tableName=table_ab9942373e884c1490e9c1fb31e1b23b, message={"server":"","servicePrincipal":"","db":"default","table":"table_ab9942373e884c1490e9c1fb31e1b23b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ab9942373e884c1490e9c1fb31e1b23b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1756887157},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/warehouse/table_ab9942373e884c1490e9c1fb31e1b23b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ab9942373e884c1490e9c1fb31e1b23b\",\"kudu.table_id\":\"53236081245549cdbf54ad2a3c947bff\",\"transient_lastDdlTime\":\"1756887157\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"fbe02324d9d04448be7fb82999004b09\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1756887157,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250903 08:13:08.091730 15566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 43, Time since last communication: 0.000s
@ 0x7f81195b9dcd google::LogMessage::Fail() at ??:0
@ 0x7f81195bdb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f81195b97cc google::LogMessage::Flush() at ??:0
W20250903 08:13:08.099330 15340 consensus_peers.cc:489] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b -> Peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Couldn't send request to peer dba49201780c41ebb58a4c8c0ae7bb1e. Status: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f81195b9a11 google::LogMessage::~LogMessage() at ??:0
I20250903 08:13:08.122735 15544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9eecf53ccf4633bc701ff21e64ab0b. Latest consensus state: current_term: 7 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:08.123006 15544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f812f9fc7b4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250903 08:13:08.126163 15544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:08.126358 15544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f812f9fb061 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250903 08:13:08.131868 15354 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b: Sending DeleteTablet for 3 replicas of tablet d1bd9a796f9947acb875272cf4e22c21
I20250903 08:13:08.134289 15354 catalog_manager.cc:6652] ReplaceTablet: tablet d1bd9a796f9947acb875272cf4e22c21 deleted and replaced by tablet 64ccb4b8673640098c29024181ce4498
@ 0x7f812fa02cdc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250903 08:13:08.134815 15354 master_service.cc:948] ReplaceTablet: received request to replace tablet 879297b2be2c4717a747a95140139e6f from {username='slave'} at 127.0.0.1:38696
I20250903 08:13:08.137939 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } } }
I20250903 08:13:08.138356 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9eecf53ccf4633bc701ff21e64ab0b [sys.catalog]: This master's current role is: LEADER
I20250903 08:13:08.149116 15354 master_service.cc:948] ReplaceTablet: received request to replace tablet da45d19c545a45c2abff9c05d8c0b293 from {username='slave'} at 127.0.0.1:38696
I20250903 08:13:08.149623 15354 master_service.cc:948] ReplaceTablet: received request to replace tablet bdfe060848fc4e9d9534e539f39137da from {username='slave'} at 127.0.0.1:38696
I20250903 08:13:08.150005 15354 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a67d22967884a02a6a89937065d2569 from {username='slave'} at 127.0.0.1:38696
I20250903 08:13:08.150595 15354 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38696:
name: "default.Table_58f55395478b4968a5f5e04b0139ac01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f812fa02b59 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
I20250903 08:13:08.168707 13871 tablet_service.cc:1515] Processing DeleteTablet for tablet d1bd9a796f9947acb875272cf4e22c21 with delete_type TABLET_DATA_DELETED (replaced by tablet 64ccb4b8673640098c29024181ce4498) from {username='slave'} at 127.0.0.1:34502
I20250903 08:13:08.169548 13870 tablet_service.cc:1468] Processing CreateTablet for tablet f321c10ca8ce4917b3ccb2ef22d80bca (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.169986 14141 tablet_service.cc:1515] Processing DeleteTablet for tablet d1bd9a796f9947acb875272cf4e22c21 with delete_type TABLET_DATA_DELETED (replaced by tablet 64ccb4b8673640098c29024181ce4498) from {username='slave'} at 127.0.0.1:42060
I20250903 08:13:08.170061 13870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321c10ca8ce4917b3ccb2ef22d80bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.171362 14139 tablet_service.cc:1468] Processing CreateTablet for tablet a88988ebf6f04d55be358560c00ca932 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.171823 14139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88988ebf6f04d55be358560c00ca932. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f811c3222b9 kudu::Thread::SuperviseThread() at ??:0
I20250903 08:13:08.173614 14140 tablet_service.cc:1468] Processing CreateTablet for tablet f321c10ca8ce4917b3ccb2ef22d80bca (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.173996 14140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321c10ca8ce4917b3ccb2ef22d80bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.177824 14142 tablet_service.cc:1468] Processing CreateTablet for tablet 64ccb4b8673640098c29024181ce4498 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.178234 14142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ccb4b8673640098c29024181ce4498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.181167 14137 tablet_service.cc:1468] Processing CreateTablet for tablet a578aed41b9d4b119932d6c4b4ea80de (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.181705 14137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a578aed41b9d4b119932d6c4b4ea80de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.181931 14138 tablet_service.cc:1468] Processing CreateTablet for tablet cf5e4ade871e4e8fba678ffe244b2f83 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.182389 14138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5e4ade871e4e8fba678ffe244b2f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.183163 13869 tablet_service.cc:1468] Processing CreateTablet for tablet 64ccb4b8673640098c29024181ce4498 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.183597 13869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ccb4b8673640098c29024181ce4498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.184327 14006 tablet_service.cc:1468] Processing CreateTablet for tablet f321c10ca8ce4917b3ccb2ef22d80bca (DEFAULT_TABLE table=default.table_c04a578b55684bd69f07a226fc5dfdfb [id=d192f9b702dc4ca092c557aa48bdaa12]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f811f7466db start_thread at ??:0
I20250903 08:13:08.184605 14004 tablet_service.cc:1468] Processing CreateTablet for tablet cf5e4ade871e4e8fba678ffe244b2f83 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.184813 14006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f321c10ca8ce4917b3ccb2ef22d80bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.185117 14004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5e4ade871e4e8fba678ffe244b2f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.185447 14007 tablet_service.cc:1468] Processing CreateTablet for tablet 64ccb4b8673640098c29024181ce4498 (DEFAULT_TABLE table=default.table_fc56d0f82c054a9facf698e29514b8d4 [id=259819c0217c4eeaa14883ebe0eea9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.185673 14005 tablet_service.cc:1468] Processing CreateTablet for tablet a88988ebf6f04d55be358560c00ca932 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.185899 14007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ccb4b8673640098c29024181ce4498. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f811846a71f clone at ??:0
I20250903 08:13:08.186156 14005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88988ebf6f04d55be358560c00ca932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.186203 14008 tablet_service.cc:1515] Processing DeleteTablet for tablet d1bd9a796f9947acb875272cf4e22c21 with delete_type TABLET_DATA_DELETED (replaced by tablet 64ccb4b8673640098c29024181ce4498) from {username='slave'} at 127.0.0.1:48624
I20250903 08:13:08.189659 14003 tablet_service.cc:1468] Processing CreateTablet for tablet a578aed41b9d4b119932d6c4b4ea80de (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.190387 14003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a578aed41b9d4b119932d6c4b4ea80de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.194232 13868 tablet_service.cc:1468] Processing CreateTablet for tablet a88988ebf6f04d55be358560c00ca932 (DEFAULT_TABLE table=default.table_517de34d608c42cc9d4aeafea81f283f [id=7fe91d7771f94773b9c3473c7e97a9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.194723 13868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88988ebf6f04d55be358560c00ca932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.197746 13866 tablet_service.cc:1468] Processing CreateTablet for tablet a578aed41b9d4b119932d6c4b4ea80de (DEFAULT_TABLE table=default.table_feb51f00a18446ed842cbf54db2f3e06 [id=44404f8a2b6340aba0f9e1eb32d7978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.198244 13866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a578aed41b9d4b119932d6c4b4ea80de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.201670 13867 tablet_service.cc:1468] Processing CreateTablet for tablet cf5e4ade871e4e8fba678ffe244b2f83 (DEFAULT_TABLE table=default.table_b21f9eea82374d75a52c4b6787afdf48 [id=d765a6a9d2bb462c92920cebf57d5995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:13:08.202354 13867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5e4ade871e4e8fba678ffe244b2f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:13:08.215408 15590 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:13:08.217563 15590 tablet_bootstrap.cc:654] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.227463 15590 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:13:08.227640 15590 ts_tablet_manager.cc:1397] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250903 08:13:08.228516 15590 raft_consensus.cc:357] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.228749 15590 raft_consensus.cc:383] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.228849 15590 raft_consensus.cc:738] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.231225 15590 consensus_queue.cc:260] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.233177 15590 ts_tablet_manager.cc:1428] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250903 08:13:08.233507 15590 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:13:08.235960 15590 tablet_bootstrap.cc:654] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.247292 13937 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:08.250453 15504 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" instance_seqno: 1756887155902525) as {username='slave'} at 127.12.250.65:60333; Asking this server to re-register.
I20250903 08:13:08.251829 13937 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:08.253662 15586 tablet_replica.cc:331] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: stopping tablet replica
I20250903 08:13:08.253885 15586 raft_consensus.cc:2241] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:13:08.254402 15586 raft_consensus.cc:2270] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:13:08.255115 15586 ts_tablet_manager.cc:1905] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:13:08.256209 15504 ts_manager.cc:194] Registered new tserver with Master: 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:13:08.257400 15588 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:13:08.259439 15588 tablet_bootstrap.cc:654] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.260434 15587 tablet_replica.cc:331] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: stopping tablet replica
I20250903 08:13:08.260649 15587 raft_consensus.cc:2241] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:13:08.260923 15587 raft_consensus.cc:2270] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:13:08.261562 15587 ts_tablet_manager.cc:1905] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 08:13:08.264813 13944 tablet.cc:2378] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:13:08.278383 15588 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:13:08.278560 15588 ts_tablet_manager.cc:1397] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20250903 08:13:08.279450 15588 raft_consensus.cc:357] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.279664 15588 raft_consensus.cc:383] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.279742 15588 raft_consensus.cc:738] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.283874 15590 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:13:08.284062 15590 ts_tablet_manager.cc:1397] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.008s
I20250903 08:13:08.284938 15590 raft_consensus.cc:357] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.285156 15590 raft_consensus.cc:383] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.285230 15590 raft_consensus.cc:738] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.285490 15590 consensus_queue.cc:260] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.285842 15587 ts_tablet_manager.cc:1918] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:13:08.285949 15590 ts_tablet_manager.cc:1428] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:13:08.285984 15587 log.cc:1199] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-2/wal/wals/d1bd9a796f9947acb875272cf4e22c21
I20250903 08:13:08.286293 15590 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:13:08.286437 15587 ts_tablet_manager.cc:1939] T d1bd9a796f9947acb875272cf4e22c21 P 0b68b07307444fdebbee0bff3fb79869: Deleting consensus metadata
I20250903 08:13:08.288508 15586 ts_tablet_manager.cc:1918] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:13:08.288653 15586 log.cc:1199] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-0/wal/wals/d1bd9a796f9947acb875272cf4e22c21
I20250903 08:13:08.289124 15586 ts_tablet_manager.cc:1939] T d1bd9a796f9947acb875272cf4e22c21 P 70c467196f6846b78b5a7c7ac8a2787d: Deleting consensus metadata
I20250903 08:13:08.289422 15590 tablet_bootstrap.cc:654] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.308630 14208 heartbeater.cc:344] Connected to a master server at 127.12.250.126:42877
I20250903 08:13:08.311344 15504 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" instance_seqno: 1756887156901466) as {username='slave'} at 127.12.250.67:41129; Asking this server to re-register.
I20250903 08:13:08.312155 14208 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:08.314684 15504 ts_manager.cc:194] Registered new tserver with Master: 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.317148 15589 tablet_replica.cc:331] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: stopping tablet replica
I20250903 08:13:08.317405 15597 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:13:08.317566 15588 consensus_queue.cc:260] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.318192 15588 ts_tablet_manager.cc:1428] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.040s user 0.000s sys 0.002s
I20250903 08:13:08.318921 15588 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:13:08.320705 15588 tablet_bootstrap.cc:654] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.321604 15597 tablet_bootstrap.cc:654] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.322417 15590 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:13:08.322561 15590 ts_tablet_manager.cc:1397] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.008s
I20250903 08:13:08.317404 15589 raft_consensus.cc:2241] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 08:13:08.323014 15589 raft_consensus.cc:2270] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:13:08.323618 15589 ts_tablet_manager.cc:1905] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 08:13:08.323532 15590 raft_consensus.cc:357] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.323770 15590 raft_consensus.cc:383] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.323871 15590 raft_consensus.cc:738] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.324155 15590 consensus_queue.cc:260] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.324291 15504 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" instance_seqno: 1756887156415898) as {username='slave'} at 127.12.250.66:54485; Asking this server to re-register.
I20250903 08:13:08.325024 14074 heartbeater.cc:461] Registering TS with master...
I20250903 08:13:08.326694 15590 ts_tablet_manager.cc:1428] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250903 08:13:08.326959 15503 ts_manager.cc:194] Registered new tserver with Master: 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745)
I20250903 08:13:08.328042 15590 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:13:08.329753 15590 tablet_bootstrap.cc:654] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.330237 15589 ts_tablet_manager.cc:1918] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 08:13:08.330358 15589 log.cc:1199] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting WAL directory at /tmp/dist-test-taskVDqtkV/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1756887139440752-13289-0/minicluster-data/ts-1/wal/wals/d1bd9a796f9947acb875272cf4e22c21
I20250903 08:13:08.330824 15589 ts_tablet_manager.cc:1939] T d1bd9a796f9947acb875272cf4e22c21 P 668be13bcd854aa8afa97d2bbaf067a7: Deleting consensus metadata
I20250903 08:13:08.337111 15597 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:13:08.337291 15588 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:13:08.337306 15597 ts_tablet_manager.cc:1397] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.002s
I20250903 08:13:08.337451 15588 ts_tablet_manager.cc:1397] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.006s
I20250903 08:13:08.338138 15588 raft_consensus.cc:357] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.338343 15588 raft_consensus.cc:383] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.338418 15588 raft_consensus.cc:738] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.338670 15588 consensus_queue.cc:260] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.339213 15588 ts_tablet_manager.cc:1428] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 08:13:08.339476 15590 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:13:08.339658 15590 ts_tablet_manager.cc:1397] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20250903 08:13:08.340497 15590 raft_consensus.cc:357] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.340607 15588 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:13:08.340690 15590 raft_consensus.cc:383] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.340766 15590 raft_consensus.cc:738] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.341042 15590 consensus_queue.cc:260] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.342358 15588 tablet_bootstrap.cc:654] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.342906 15590 ts_tablet_manager.cc:1428] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250903 08:13:08.343263 15590 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d: Bootstrap starting.
I20250903 08:13:08.345314 15590 tablet_bootstrap.cc:654] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.338155 15597 raft_consensus.cc:357] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.348784 15597 raft_consensus.cc:383] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.348961 15597 raft_consensus.cc:738] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.349365 15597 consensus_queue.cc:260] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.354372 15590 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d: No bootstrap required, opened a new log
I20250903 08:13:08.354557 15590 ts_tablet_manager.cc:1397] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250903 08:13:08.355598 15590 raft_consensus.cc:357] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.355839 15590 raft_consensus.cc:383] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.355940 15590 raft_consensus.cc:738] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.356195 15590 consensus_queue.cc:260] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.356865 15590 ts_tablet_manager.cc:1428] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250903 08:13:08.358369 15588 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:13:08.358526 15588 ts_tablet_manager.cc:1397] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20250903 08:13:08.359382 15588 raft_consensus.cc:357] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.359579 15588 raft_consensus.cc:383] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.359663 15588 raft_consensus.cc:738] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.359895 15588 consensus_queue.cc:260] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.360482 15588 ts_tablet_manager.cc:1428] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 08:13:08.360798 15588 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:13:08.360806 15597 ts_tablet_manager.cc:1428] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250903 08:13:08.361159 15597 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:13:08.362593 15588 tablet_bootstrap.cc:654] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.367525 15597 tablet_bootstrap.cc:654] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.375176 15597 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:13:08.375176 15588 tablet_bootstrap.cc:492] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:13:08.375360 15597 ts_tablet_manager.cc:1397] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250903 08:13:08.375360 15588 ts_tablet_manager.cc:1397] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250903 08:13:08.376153 15597 raft_consensus.cc:357] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.376154 15588 raft_consensus.cc:357] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.376363 15588 raft_consensus.cc:383] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.376363 15597 raft_consensus.cc:383] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.376451 15588 raft_consensus.cc:738] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.376461 15597 raft_consensus.cc:738] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.376739 15597 consensus_queue.cc:260] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.376740 15588 consensus_queue.cc:260] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.377298 15597 ts_tablet_manager.cc:1428] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:13:08.379312 15597 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:13:08.381153 15597 tablet_bootstrap.cc:654] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.383908 15597 tablet_bootstrap.cc:492] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:13:08.384056 15597 ts_tablet_manager.cc:1397] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250903 08:13:08.384828 15597 raft_consensus.cc:357] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.385020 15597 raft_consensus.cc:383] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.385128 15597 raft_consensus.cc:738] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.385411 15597 consensus_queue.cc:260] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.385936 15597 ts_tablet_manager.cc:1428] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:13:08.386224 15597 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:13:08.387971 15597 tablet_bootstrap.cc:654] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.389528 15588 ts_tablet_manager.cc:1428] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 08:13:08.391985 15588 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: Bootstrap starting.
I20250903 08:13:08.393909 15588 tablet_bootstrap.cc:654] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.397432 15588 tablet_bootstrap.cc:492] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: No bootstrap required, opened a new log
I20250903 08:13:08.397617 15588 ts_tablet_manager.cc:1397] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250903 08:13:08.398639 15588 raft_consensus.cc:357] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.398893 15588 raft_consensus.cc:383] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.398991 15588 raft_consensus.cc:738] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.399282 15588 consensus_queue.cc:260] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.400832 15597 tablet_bootstrap.cc:492] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:13:08.401005 15597 ts_tablet_manager.cc:1397] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250903 08:13:08.401815 15597 raft_consensus.cc:357] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.402005 15597 raft_consensus.cc:383] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.402096 15597 raft_consensus.cc:738] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.402377 15597 consensus_queue.cc:260] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.402885 15597 ts_tablet_manager.cc:1428] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:13:08.403227 15597 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7: Bootstrap starting.
I20250903 08:13:08.406322 15588 ts_tablet_manager.cc:1428] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 08:13:08.409426 15597 tablet_bootstrap.cc:654] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7: Neither blocks nor log segments found. Creating new log.
I20250903 08:13:08.417476 15597 tablet_bootstrap.cc:492] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7: No bootstrap required, opened a new log
I20250903 08:13:08.417652 15597 ts_tablet_manager.cc:1397] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20250903 08:13:08.418574 15597 raft_consensus.cc:357] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.418804 15597 raft_consensus.cc:383] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:13:08.418905 15597 raft_consensus.cc:738] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Initialized, Role: FOLLOWER
I20250903 08:13:08.419219 15597 consensus_queue.cc:260] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.419759 15597 ts_tablet_manager.cc:1428] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:13:08.483253 15584 raft_consensus.cc:491] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:08.483470 15584 raft_consensus.cc:513] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.484462 15584 leader_election.cc:290] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.485127 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5e4ade871e4e8fba678ffe244b2f83" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:13:08.485289 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5e4ade871e4e8fba678ffe244b2f83" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:13:08.485392 13890 raft_consensus.cc:2466] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:13:08.485515 14162 raft_consensus.cc:2466] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:13:08.485956 13964 leader_election.cc:304] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.486315 15584 raft_consensus.cc:2802] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:13:08.486431 15584 raft_consensus.cc:491] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:08.486510 15584 raft_consensus.cc:3058] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.489410 15584 raft_consensus.cc:513] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.490618 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5e4ade871e4e8fba678ffe244b2f83" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:13:08.490620 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5e4ade871e4e8fba678ffe244b2f83" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:13:08.490814 14162 raft_consensus.cc:3058] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.490844 13890 raft_consensus.cc:3058] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.493508 14162 raft_consensus.cc:2466] T cf5e4ade871e4e8fba678ffe244b2f83 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:13:08.493860 13890 raft_consensus.cc:2466] T cf5e4ade871e4e8fba678ffe244b2f83 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
W20250903 08:13:08.493911 14212 tablet.cc:2378] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:13:08.494877 13964 leader_election.cc:304] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.495275 15584 leader_election.cc:290] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.495337 15572 raft_consensus.cc:491] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:08.495491 15584 raft_consensus.cc:2802] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:13:08.495492 15572 raft_consensus.cc:513] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.495751 15584 raft_consensus.cc:695] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Becoming Leader. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Running, Role: LEADER
I20250903 08:13:08.496081 15584 consensus_queue.cc:237] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.496457 15572 leader_election.cc:290] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.496929 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a578aed41b9d4b119932d6c4b4ea80de" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:13:08.497126 14162 raft_consensus.cc:2466] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:13:08.497571 13962 leader_election.cc:304] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:13:08.498230 15572 raft_consensus.cc:2802] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:13:08.498339 15572 raft_consensus.cc:491] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:08.498422 15572 raft_consensus.cc:3058] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.502238 15572 raft_consensus.cc:513] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.504113 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a578aed41b9d4b119932d6c4b4ea80de" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:13:08.504330 13889 raft_consensus.cc:3058] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.504513 15572 leader_election.cc:290] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.504113 13890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a578aed41b9d4b119932d6c4b4ea80de" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:13:08.505664 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a578aed41b9d4b119932d6c4b4ea80de" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:13:08.505895 14162 raft_consensus.cc:3058] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
W20250903 08:13:08.507452 14076 tablet.cc:2378] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 08:13:08.507841 13889 raft_consensus.cc:2466] T a578aed41b9d4b119932d6c4b4ea80de P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:13:08.507926 14162 raft_consensus.cc:2466] T a578aed41b9d4b119932d6c4b4ea80de P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:13:08.508383 13962 leader_election.cc:304] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:13:08.508636 15572 raft_consensus.cc:2802] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:13:08.508769 15572 raft_consensus.cc:695] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Becoming Leader. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Running, Role: LEADER
I20250903 08:13:08.509093 15572 consensus_queue.cc:237] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
W20250903 08:13:08.518883 13961 connection.cc:537] server connection from 127.0.0.1:48624 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.518877 13825 connection.cc:537] server connection from 127.0.0.1:34502 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.518855 14098 connection.cc:537] server connection from 127.0.0.1:42060 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.519032 14214 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.519140 13961 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.519590 13823 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.519737 14214 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.250.124:39503)
W20250903 08:13:08.520303 14214 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.250.124:39503)
W20250903 08:13:08.520604 14214 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.250.124:39503)
W20250903 08:13:08.520889 14095 connection.cc:537] client connection to 127.12.250.124:39503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 08:13:08.522658 14214 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.250.124:39503)
I20250903 08:13:08.559090 14960 raft_consensus.cc:491] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:08.559286 14960 raft_consensus.cc:513] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.560312 14960 leader_election.cc:290] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:13:08.560873 14026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ccb4b8673640098c29024181ce4498" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:13:08.561007 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ccb4b8673640098c29024181ce4498" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:13:08.561203 14026 raft_consensus.cc:2466] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:13:08.561233 13889 raft_consensus.cc:2466] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 0.
I20250903 08:13:08.561702 14097 leader_election.cc:304] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:13:08.562003 14960 raft_consensus.cc:2802] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:13:08.562143 14960 raft_consensus.cc:491] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:08.562232 14960 raft_consensus.cc:3058] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.564121 14960 raft_consensus.cc:513] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.564671 14960 leader_election.cc:290] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323)
I20250903 08:13:08.565122 14026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ccb4b8673640098c29024181ce4498" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:13:08.565132 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64ccb4b8673640098c29024181ce4498" candidate_uuid: "0b68b07307444fdebbee0bff3fb79869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:13:08.565322 14026 raft_consensus.cc:3058] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.565341 13889 raft_consensus.cc:3058] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.567044 13889 raft_consensus.cc:2466] T 64ccb4b8673640098c29024181ce4498 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:13:08.567150 14026 raft_consensus.cc:2466] T 64ccb4b8673640098c29024181ce4498 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b68b07307444fdebbee0bff3fb79869 in term 1.
I20250903 08:13:08.567457 14098 leader_election.cc:304] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.567802 14960 raft_consensus.cc:2802] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:13:08.568313 14960 raft_consensus.cc:695] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [term 1 LEADER]: Becoming Leader. State: Replica: 0b68b07307444fdebbee0bff3fb79869, State: Running, Role: LEADER
I20250903 08:13:08.568917 14960 consensus_queue.cc:237] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } } peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } }
I20250903 08:13:08.597816 15467 raft_consensus.cc:491] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:08.598016 15467 raft_consensus.cc:513] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.598901 15467 leader_election.cc:290] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.599363 14026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f321c10ca8ce4917b3ccb2ef22d80bca" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7" is_pre_election: true
I20250903 08:13:08.599398 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f321c10ca8ce4917b3ccb2ef22d80bca" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:13:08.599581 14026 raft_consensus.cc:2466] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:13:08.599594 14162 raft_consensus.cc:2466] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 0.
I20250903 08:13:08.600005 13825 leader_election.cc:304] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.600281 15467 raft_consensus.cc:2802] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:13:08.600386 15467 raft_consensus.cc:491] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:08.600467 15467 raft_consensus.cc:3058] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.602481 15467 raft_consensus.cc:513] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.603006 15467 leader_election.cc:290] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Requested vote from peers 668be13bcd854aa8afa97d2bbaf067a7 (127.12.250.66:42745), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.603504 14026 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f321c10ca8ce4917b3ccb2ef22d80bca" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668be13bcd854aa8afa97d2bbaf067a7"
I20250903 08:13:08.603536 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f321c10ca8ce4917b3ccb2ef22d80bca" candidate_uuid: "70c467196f6846b78b5a7c7ac8a2787d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:13:08.603688 14026 raft_consensus.cc:3058] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.603747 14162 raft_consensus.cc:3058] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.605742 14162 raft_consensus.cc:2466] T f321c10ca8ce4917b3ccb2ef22d80bca P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:13:08.605808 14026 raft_consensus.cc:2466] T f321c10ca8ce4917b3ccb2ef22d80bca P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c467196f6846b78b5a7c7ac8a2787d in term 1.
I20250903 08:13:08.606235 13825 leader_election.cc:304] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.606603 15467 raft_consensus.cc:2802] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:13:08.607028 15467 raft_consensus.cc:695] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [term 1 LEADER]: Becoming Leader. State: Replica: 70c467196f6846b78b5a7c7ac8a2787d, State: Running, Role: LEADER
I20250903 08:13:08.607347 15467 consensus_queue.cc:237] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.664460 15623 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:08.664757 15623 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:08.666752 15623 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:08.667666 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:08.668941 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:08.669184 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:08.669391 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:08.669777 15623 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:08.848446 15572 raft_consensus.cc:491] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:13:08.848686 15572 raft_consensus.cc:513] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.849380 15572 leader_election.cc:290] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.849943 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a88988ebf6f04d55be358560c00ca932" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d" is_pre_election: true
I20250903 08:13:08.849996 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a88988ebf6f04d55be358560c00ca932" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869" is_pre_election: true
I20250903 08:13:08.850244 13889 raft_consensus.cc:2466] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:13:08.850244 14162 raft_consensus.cc:2466] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 0.
I20250903 08:13:08.850726 13964 leader_election.cc:304] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7, 70c467196f6846b78b5a7c7ac8a2787d; no voters:
I20250903 08:13:08.851058 15572 raft_consensus.cc:2802] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:13:08.851140 15572 raft_consensus.cc:491] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:13:08.851213 15572 raft_consensus.cc:3058] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.853020 15572 raft_consensus.cc:513] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:08.853579 15572 leader_election.cc:290] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Requested vote from peers 70c467196f6846b78b5a7c7ac8a2787d (127.12.250.65:45323), 0b68b07307444fdebbee0bff3fb79869 (127.12.250.67:34257)
I20250903 08:13:08.854101 14162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a88988ebf6f04d55be358560c00ca932" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b68b07307444fdebbee0bff3fb79869"
I20250903 08:13:08.854171 13889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a88988ebf6f04d55be358560c00ca932" candidate_uuid: "668be13bcd854aa8afa97d2bbaf067a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c467196f6846b78b5a7c7ac8a2787d"
I20250903 08:13:08.854323 14162 raft_consensus.cc:3058] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.854436 13889 raft_consensus.cc:3058] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:13:08.856376 14162 raft_consensus.cc:2466] T a88988ebf6f04d55be358560c00ca932 P 0b68b07307444fdebbee0bff3fb79869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:13:08.856353 13889 raft_consensus.cc:2466] T a88988ebf6f04d55be358560c00ca932 P 70c467196f6846b78b5a7c7ac8a2787d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 668be13bcd854aa8afa97d2bbaf067a7 in term 1.
I20250903 08:13:08.856796 13962 leader_election.cc:304] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b68b07307444fdebbee0bff3fb79869, 668be13bcd854aa8afa97d2bbaf067a7; no voters:
I20250903 08:13:08.857062 15572 raft_consensus.cc:2802] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:13:08.857357 15572 raft_consensus.cc:695] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [term 1 LEADER]: Becoming Leader. State: Replica: 668be13bcd854aa8afa97d2bbaf067a7, State: Running, Role: LEADER
I20250903 08:13:08.857739 15572 consensus_queue.cc:237] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 } } peers { permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 } } peers { permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 } }
I20250903 08:13:09.384586 15636 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:09.384824 15636 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:09.385601 15636 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:09.387307 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:09.387514 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:09.387666 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:09.387935 15636 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:09.422086 15633 consensus_queue.cc:1035] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.427615 15643 consensus_queue.cc:1035] T 64ccb4b8673640098c29024181ce4498 P 0b68b07307444fdebbee0bff3fb79869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.433193 15467 consensus_queue.cc:1035] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "668be13bcd854aa8afa97d2bbaf067a7" member_type: VOTER last_known_addr { host: "127.12.250.66" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.439430 15467 consensus_queue.cc:1035] T f321c10ca8ce4917b3ccb2ef22d80bca P 70c467196f6846b78b5a7c7ac8a2787d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.460031 15572 consensus_queue.cc:1035] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.467571 15651 consensus_queue.cc:1035] T cf5e4ade871e4e8fba678ffe244b2f83 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.489939 15651 consensus_queue.cc:1035] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.495751 15651 consensus_queue.cc:1035] T a578aed41b9d4b119932d6c4b4ea80de P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.648538 15651 consensus_queue.cc:1035] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c467196f6846b78b5a7c7ac8a2787d" member_type: VOTER last_known_addr { host: "127.12.250.65" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.654718 15572 consensus_queue.cc:1035] T a88988ebf6f04d55be358560c00ca932 P 668be13bcd854aa8afa97d2bbaf067a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b68b07307444fdebbee0bff3fb79869" member_type: VOTER last_known_addr { host: "127.12.250.67" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:13:09.891281 15659 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:09.891448 15659 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:09.891991 15659 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:09.893808 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:09.893960 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:09.894120 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:09.894397 15659 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:10.354462 13289 master-stress-test.cc:577] Killing test thread 1/25
I20250903 08:13:11.375005 15686 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:11.375202 15686 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:11.375825 15686 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:11.377806 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:11.377957 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:11.378118 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:11.378378 15686 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:12.712783 15705 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:12.713011 15705 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:12.713706 15705 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:12.715822 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:12.715982 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:12.716094 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:12.716331 15705 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:12.893697 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:12.893916 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:15.297623 15743 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:15.297808 15743 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:15.298446 15743 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:15.299374 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250903 08:13:15.300489 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:15.300626 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:15.300761 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:15.301035 15743 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:17.896137 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:17.896313 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:20.877686 15818 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:20.877889 15818 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:20.878599 15818 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:20.879563 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:13:20.880707 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:20.880914 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:20.881024 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:20.881292 15818 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:22.898952 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:22.899139 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:27.512490 15903 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:27.512699 15903 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:27.513391 15903 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:27.514284 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:13:27.515807 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:27.515950 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:13:27.516084 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:27.516331 15903 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:27.901366 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:27.901541 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:31.373939 15961 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:31.374198 15961 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:31.374963 15961 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:31.376832 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:31.376981 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:31.377156 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:31.377401 15961 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:32.904165 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:32.904340 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:36.600420 16034 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:36.600644 16034 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:36.601300 16034 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:36.602226 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:13:36.603467 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:36.603623 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:36.603732 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:36.603966 16034 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250903 08:13:37.592278 13938 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (104 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:37.832418 14209 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (105 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:13:37.852586 14075 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (105 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:37.906879 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:37.907044 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:40.270428 16081 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:40.270642 16081 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:40.271279 16081 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:40.273545 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:40.273708 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:40.273850 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:40.274098 16081 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:13:42.909518 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:42.909689 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:47.912061 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:47.912269 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:52.915311 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:52.915488 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:57.918179 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:13:57.918334 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:13:59.296185 16334 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:13:59.296406 16334 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:13:59.297278 16334 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:13:59.298247 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:13:59.299430 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:13:59.299582 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:13:59.299693 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:13:59.299916 16334 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:02.357448 16375 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:14:02.357647 16375 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:14:02.358314 16375 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:14:02.360355 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:14:02.360512 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:14:02.360641 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:14:02.360906 16375 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:02.920874 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:02.921051 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:07.923949 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:07.924144 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:12.927173 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:12.927362 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:16.766469 16558 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:14:16.766688 16558 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:14:16.767346 16558 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:14:16.768275 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:14:16.769627 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:14:16.769769 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:14:16.769906 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:14:16.770171 16558 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:17.931509 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:17.931700 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:22.934615 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:22.934839 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:27.937770 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:27.937953 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:31.049727 16748 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:14:31.049948 16748 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:14:31.051151 16748 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:14:31.052124 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:14:31.053177 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:14:31.053360 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:14:31.053493 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:14:31.053747 16748 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:32.941344 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:32.941498 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:14:37.779204 13938 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (224 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:14:37.944201 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:37.944434 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:14:38.020361 14207 heartbeater.cc:646] Failed to heartbeat to 127.12.250.124:39503 (190 consecutive failures): Network error: Failed to ping master at 127.12.250.124:39503: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:14:38.033126 14075 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (225 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:14:41.077175 16880 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:14:41.077435 16880 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:14:41.078130 16880 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:14:41.079048 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:14:41.080025 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:14:41.080163 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:14:41.080308 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:14:41.080515 16880 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:42.946856 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:42.947006 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:47.949528 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:47.949687 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:52.952667 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:52.952862 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:14:57.712548 17097 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:14:57.712779 17097 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:14:57.713650 17097 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:14:57.714761 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:14:57.716032 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:14:57.716198 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:14:57.716285 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:14:57.716470 17097 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:14:57.955401 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:14:57.955567 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:00.213985 17132 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:00.214166 17132 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:00.214851 17132 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:00.216590 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:00.216759 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:00.216917 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:00.217136 17132 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:02.958240 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:02.958415 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:03.436642 17178 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:03.436843 17178 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:03.437557 17178 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:03.438198 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:15:03.439325 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:03.439458 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:03.439553 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:03.439796 17178 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:06.438962 17219 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:06.439158 17219 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:06.440250 17219 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:06.442111 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:06.442273 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:06.442417 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:06.442665 17219 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:07.937572 15493 maintenance_manager.cc:419] P 1e5331161d9e4df39004183d79dc3611: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033355
I20250903 08:15:07.966315 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:07.966493 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:07.983417 15492 maintenance_manager.cc:643] P 1e5331161d9e4df39004183d79dc3611: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.019s sys 0.005s Metrics: {"bytes_written":15055,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":776,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":32,"thread_start_us":470,"threads_started":2,"wal-append.queue_time_us":294}
I20250903 08:15:07.984513 15493 maintenance_manager.cc:419] P 1e5331161d9e4df39004183d79dc3611: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1659 bytes on disk
I20250903 08:15:07.985122 15492 maintenance_manager.cc:643] P 1e5331161d9e4df39004183d79dc3611: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20250903 08:15:12.969014 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:12.969170 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:17.971709 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:17.971868 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:22.974673 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:22.974860 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:24.862653 17486 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:24.862856 17486 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:24.863683 17486 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:24.864614 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:15:24.865774 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:15:24.865964 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:15:24.866073 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:24.866287 17486 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:27.977357 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:27.977563 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:31.882532 17604 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:31.882712 17604 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:31.883242 17604 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:31.884141 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:15:31.885109 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:15:31.885236 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:15:31.885388 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:31.885596 17604 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:32.979902 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:32.980063 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:15:37.967408 13936 heartbeater.cc:646] Failed to heartbeat to 127.12.250.124:39503 (309 consecutive failures): Network error: Failed to ping master at 127.12.250.124:39503: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:37.982427 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:37.982571 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:15:38.251135 14075 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (345 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:38.252624 14207 heartbeater.cc:646] Failed to heartbeat to 127.12.250.124:39503 (310 consecutive failures): Network error: Failed to ping master at 127.12.250.124:39503: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:39.190871 17718 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:39.191066 17718 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:39.191663 17718 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:39.193229 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:15:39.194422 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:39.194598 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:39.194712 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:39.194969 17718 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:42.984722 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:42.984934 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:47.987744 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:47.987933 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:52.990944 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:52.991110 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:15:55.685364 17953 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:15:55.685578 17953 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:15:55.686350 17953 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:15:55.687693 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:15:55.688884 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:15:55.689038 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:15:55.689160 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:15:55.689395 17953 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:15:57.993826 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:15:57.993988 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:02.996410 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:02.996572 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:07.101985 18140 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:16:07.102173 18140 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:16:07.102857 18140 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:16:07.103681 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:16:07.105068 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:16:07.105197 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:16:07.105327 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:16:07.105558 18140 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:16:07.999187 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:07.999351 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:13.002070 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:13.002246 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:18.004865 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:18.005039 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:23.007736 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:23.007901 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:25.797998 18384 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:16:25.798164 18384 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:16:25.798843 18384 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:16:25.799755 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:16:25.800887 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:16:25.801033 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:16:25.801133 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:16:25.801411 18384 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:16:28.010962 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:28.011121 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:33.013741 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:33.013927 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:38.016381 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:38.016533 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:16:38.137491 13936 heartbeater.cc:646] Failed to heartbeat to 127.12.250.124:39503 (429 consecutive failures): Network error: Failed to ping master at 127.12.250.124:39503: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:16:38.523613 14209 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (465 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:16:38.538653 14075 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (465 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:16:40.095026 18615 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:16:40.095240 18615 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:16:40.095937 18615 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:16:40.096983 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:16:40.098105 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:16:40.098279 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:16:40.098412 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:16:40.098628 18615 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:16:43.019312 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:43.019482 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:48.022353 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:48.022518 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:53.025141 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:53.025365 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:16:57.588162 18898 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:16:57.588366 18898 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:16:57.589203 18898 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:16:57.590230 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:16:57.591487 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:16:57.591670 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:16:57.591791 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:16:57.592022 18898 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:16:58.027884 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:16:58.028064 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:03.030865 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:03.031042 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:03.614078 19001 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:17:03.614269 19001 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:17:03.614969 19001 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:17:03.615725 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:17:03.616888 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:17:03.617028 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:17:03.617141 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:17:03.617376 19001 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:17:08.034123 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:08.034324 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:13.037504 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:13.037673 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:18.040165 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:18.040318 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:21.969477 19291 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:17:21.969686 19291 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:17:21.970577 19291 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:17:21.971540 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:17:21.972748 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:17:21.972973 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:17:21.973081 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:17:21.973321 19291 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:17:23.043344 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:23.043507 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:28.046118 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:28.046294 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:33.050117 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:33.050351 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:38.053325 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:38.053527 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250903 08:17:38.315224 13938 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (584 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:17:38.698272 14207 heartbeater.cc:646] Failed to heartbeat to 127.12.250.124:39503 (550 consecutive failures): Network error: Failed to ping master at 127.12.250.124:39503: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:17:38.717109 14075 heartbeater.cc:646] Failed to heartbeat to 127.12.250.125:34047 (585 consecutive failures): Network error: Failed to ping master at 127.12.250.125:34047: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:17:39.033200 19555 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:17:39.033421 19555 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:17:39.034282 19555 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:17:39.035297 15489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:17:39.036418 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
W20250903 08:17:39.036574 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
I20250903 08:17:39.036688 15489 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:17:39.036945 19555 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:17:41.496479 19597 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:17:41.496723 19597 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:17:41.497520 19597 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:17:41.499557 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:17:41.499713 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:17:41.499876 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:17:41.500115 19597 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:17:43.056226 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:43.056396 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:48.059018 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:48.059177 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:53.062542 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:53.062728 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:17:57.997594 19827 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:17:57.997809 19827 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:17:57.998618 19827 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:17:57.999575 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250903 08:17:58.000598 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:17:58.000756 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:17:58.000860 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:17:58.001139 19827 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:17:58.065232 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:17:58.065452 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:18:03.068576 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:18:03.068782 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:18:03.801019 19894 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff9eecf53ccf4633bc701ff21e64ab0b)
I20250903 08:18:03.801301 19894 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e5331161d9e4df39004183d79dc3611" member_type: VOTER last_known_addr { host: "127.12.250.126" port: 42877 } } peers { permanent_uuid: "dba49201780c41ebb58a4c8c0ae7bb1e" member_type: VOTER last_known_addr { host: "127.12.250.125" port: 34047 } } peers { permanent_uuid: "ff9eecf53ccf4633bc701ff21e64ab0b" member_type: VOTER last_known_addr { host: "127.12.250.124" port: 39503 } }
I20250903 08:18:03.801992 19894 leader_election.cc:290] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047), ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503)
W20250903 08:18:03.803545 15488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250903 08:18:03.804746 15489 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dba49201780c41ebb58a4c8c0ae7bb1e (127.12.250.125:34047): Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111)
W20250903 08:18:03.804905 15488 leader_election.cc:336] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ff9eecf53ccf4633bc701ff21e64ab0b (127.12.250.124:39503): Network error: Client connection negotiation failed: client connection to 127.12.250.124:39503: connect: Connection refused (error 111)
I20250903 08:18:03.805042 15488 leader_election.cc:304] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e5331161d9e4df39004183d79dc3611; no voters: dba49201780c41ebb58a4c8c0ae7bb1e, ff9eecf53ccf4633bc701ff21e64ab0b
I20250903 08:18:03.805325 19894 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 1e5331161d9e4df39004183d79dc3611 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250903 08:18:08.071504 15473 master_runner.cc:186] Error getting master registration for 127.12.250.125:34047: Network error: Client connection negotiation failed: client connection to 127.12.250.125:34047: connect: Connection refused (error 111),
I20250903 08:18:08.071678 15473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 08:18:08.245484 15493 maintenance_manager.cc:419] P 1e5331161d9e4df39004183d79dc3611: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1659 bytes on disk
I20250903 08:18:08.250588 15492 maintenance_manager.cc:643] P 1e5331161d9e4df39004183d79dc3611: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"dirs.queue_time_us":386,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":157,"thread_start_us":474,"threads_started":2,"wal-append.queue_time_us":320}
W20250903 08:18:08.657302 14214 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
W20250903 08:18:08.657667 14214 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
W20250903 08:18:08.657797 14214 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
W20250903 08:18:08.657928 14214 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
F20250903 08:18:08.668931 14225 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3b5331a45947402b9deafb9d63b55b7e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
*** Aborted at 1756887488 (unix time) try "date -d @1756887488" if you are using GNU date ***
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000033e9) received by PID 13289 (TID 0x7fa3abdae700) from PID 13289; stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676235 14231 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ecab4f09e824890b839f9ec45bfa3da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676235 14231 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ecab4f09e824890b839f9ec45bfa3da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677331 14229 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1fb5beb8dbab4781ab4c84eb8414a936 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676235 14231 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ecab4f09e824890b839f9ec45bfa3da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677331 14229 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1fb5beb8dbab4781ab4c84eb8414a936 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677366 14236 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676235 14231 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ecab4f09e824890b839f9ec45bfa3da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677331 14229 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1fb5beb8dbab4781ab4c84eb8414a936 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677366 14236 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677542 14234 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7fa3c170f980 (unknown) at ??:0
F20250903 08:18:08.668936 14228 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c88874dd846641f19e00bc8d13e3377b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.668936 14227 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1a3f51ccc43c47e4abaeaa2384e5f332 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.670029 14224 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6221cb658b3f49fb8d60a02df2b7a834 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671185 14246 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671487 14243 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671576 14240 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671751 14242 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671927 14233 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.671928 14232 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.672196 14226 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_94faf7a1e44247f6aeaed3759e0b4f99 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.673609 14239 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674405 14230 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_58f55395478b4968a5f5e04b0139ac01 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674566 14223 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6a11e40b85c9416abb127a1c1ecf1e0b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674418 14245 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.674700 14235 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675879 14222 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_250ab81379d64db694b3b9013da7f8a1 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.675892 14244 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676131 14238 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.676235 14231 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0ecab4f09e824890b839f9ec45bfa3da on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677331 14229 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1fb5beb8dbab4781ab4c84eb8414a936 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677366 14236 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677542 14234 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)F20250903 08:18:08.677655 14237 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.12.250.126:42877,127.12.250.125:34047,127.12.250.124:39503, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7fa3ba6d5fb7 gsignal at ??:0
@ 0x7fa3ba6d7921 abort at ??:0
@ 0x7fa3bd40ddcd google::LogMessage::Fail() at ??:0
@ 0x7fa3bd411b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa3bd40d7cc google::LogMessage::Flush() at ??:0
@ 0x7fa3bd40ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fa3bad5d6df (unknown) at ??:0
@ 0x7fa3c17046db start_thread at ??:0
@ 0x7fa3ba7b871f clone at ??:0